-- Logs begin at Tue 2022-09-13 15:29:16 UTC, end at Tue 2022-09-13 16:56:05 UTC. -- Sep 13 15:52:25.794053 dv-cinder-820027-28-iscsi systemd[1]: Starting Devstack devstack@keystone.service... Sep 13 15:52:25.802724 dv-cinder-820027-28-iscsi devstack@keystone.service[81426]: [uWSGI] getting INI configuration from /etc/keystone/keystone-uwsgi-public.ini Sep 13 15:52:25.804991 dv-cinder-820027-28-iscsi devstack@keystone.service[81426]: *** Starting uWSGI 2.0.18-debian (64bit) on [Tue Sep 13 15:52:25 2022] *** Sep 13 15:52:25.804991 dv-cinder-820027-28-iscsi devstack@keystone.service[81426]: compiled with version: 10.0.1 20200405 (experimental) [master revision 0be9efad938:fcb98e4978a:705510a708d3642c9c962beb663c476167e4e8a4] on 11 April 2020 11:15:55 Sep 13 15:52:25.804991 dv-cinder-820027-28-iscsi devstack@keystone.service[81426]: os: Linux-5.4.0-72-generic #80-Ubuntu SMP Mon Apr 12 17:35:00 UTC 2021 Sep 13 15:52:25.804991 dv-cinder-820027-28-iscsi devstack@keystone.service[81426]: nodename: dv-cinder-820027-28-iscsi Sep 13 15:52:25.804991 dv-cinder-820027-28-iscsi devstack@keystone.service[81426]: machine: x86_64 Sep 13 15:52:25.804991 dv-cinder-820027-28-iscsi devstack@keystone.service[81426]: clock source: unix Sep 13 15:52:25.804991 dv-cinder-820027-28-iscsi devstack@keystone.service[81426]: pcre jit disabled Sep 13 15:52:25.804991 dv-cinder-820027-28-iscsi devstack@keystone.service[81426]: detected number of CPU cores: 4 Sep 13 15:52:25.804991 dv-cinder-820027-28-iscsi devstack@keystone.service[81426]: current working directory: / Sep 13 15:52:25.804991 dv-cinder-820027-28-iscsi devstack@keystone.service[81426]: detected binary path: /usr/bin/uwsgi-core Sep 13 15:52:25.805525 dv-cinder-820027-28-iscsi devstack@keystone.service[81426]: your processes number limit is 47680 Sep 13 15:52:25.805525 dv-cinder-820027-28-iscsi devstack@keystone.service[81426]: your memory page size is 4096 bytes Sep 13 15:52:25.805525 dv-cinder-820027-28-iscsi devstack@keystone.service[81426]: detected max file descriptor number: 2048 Sep 13 15:52:25.805525 dv-cinder-820027-28-iscsi devstack@keystone.service[81426]: lock engine: pthread robust mutexes Sep 13 15:52:25.805525 dv-cinder-820027-28-iscsi devstack@keystone.service[81426]: thunder lock: enabled Sep 13 15:52:25.805525 dv-cinder-820027-28-iscsi devstack@keystone.service[81426]: uwsgi socket 0 bound to UNIX address /var/run/uwsgi/keystone-wsgi-public.socket fd 4 Sep 13 15:52:25.805525 dv-cinder-820027-28-iscsi devstack@keystone.service[81426]: Python version: 3.8.10 (default, Jun 22 2022, 20:18:18) [GCC 9.4.0] Sep 13 15:52:25.845314 dv-cinder-820027-28-iscsi devstack@keystone.service[81426]: Python main interpreter initialized at 0x5623371229c0 Sep 13 15:52:25.845314 dv-cinder-820027-28-iscsi devstack@keystone.service[81426]: python threads support enabled Sep 13 15:52:25.845314 dv-cinder-820027-28-iscsi devstack@keystone.service[81426]: your server socket listen backlog is limited to 100 connections Sep 13 15:52:25.845314 dv-cinder-820027-28-iscsi devstack@keystone.service[81426]: your mercy for graceful operations on workers is 90 seconds Sep 13 15:52:25.845644 dv-cinder-820027-28-iscsi devstack@keystone.service[81426]: mapped 403077 bytes (393 KB) for 2 cores Sep 13 15:52:25.845741 dv-cinder-820027-28-iscsi devstack@keystone.service[81426]: *** Operational MODE: preforking *** Sep 13 15:52:25.846010 dv-cinder-820027-28-iscsi devstack@keystone.service[81426]: *** uWSGI is running in multiple interpreter mode *** Sep 13 15:52:25.846010 dv-cinder-820027-28-iscsi devstack@keystone.service[81426]: spawned uWSGI master process (pid: 81426) Sep 13 15:52:25.846203 dv-cinder-820027-28-iscsi systemd[1]: Started Devstack devstack@keystone.service. Sep 13 15:52:25.850944 dv-cinder-820027-28-iscsi devstack@keystone.service[81426]: spawned uWSGI worker 1 (pid: 81437, cores: 1) Sep 13 15:52:25.850944 dv-cinder-820027-28-iscsi devstack@keystone.service[81426]: spawned uWSGI worker 2 (pid: 81438, cores: 1) Sep 13 15:52:25.850944 dv-cinder-820027-28-iscsi devstack@keystone.service[81426]: running "unix_signal:15 gracefully_kill_them_all" (master-start)... Sep 13 15:52:30.141576 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: /usr/lib/python3/dist-packages/secretstorage/dhcrypto.py:15: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead Sep 13 15:52:30.141576 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: from cryptography.utils import int_from_bytes Sep 13 15:52:30.142355 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: /usr/lib/python3/dist-packages/secretstorage/util.py:19: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead Sep 13 15:52:30.142355 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: from cryptography.utils import int_from_bytes Sep 13 15:52:30.143046 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: /usr/lib/python3/dist-packages/secretstorage/dhcrypto.py:15: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead Sep 13 15:52:30.143046 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: from cryptography.utils import int_from_bytes Sep 13 15:52:30.143501 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: /usr/lib/python3/dist-packages/secretstorage/util.py:19: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead Sep 13 15:52:30.143501 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: from cryptography.utils import int_from_bytes Sep 13 15:52:30.192333 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] ******************************************************************************** {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2589}} Sep 13 15:52:30.192516 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] Configuration options gathered from: {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2590}} Sep 13 15:52:30.192705 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] command line args: [] {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2591}} Sep 13 15:52:30.192980 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] config files: ['/etc/keystone/keystone.conf'] {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2592}} Sep 13 15:52:30.193159 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] ================================================================================ {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2594}} Sep 13 15:52:30.193418 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] admin_token = **** {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 13 15:52:30.193610 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] ******************************************************************************** {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2589}} Sep 13 15:52:30.193700 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] config_dir = [] {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 13 15:52:30.193809 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] Configuration options gathered from: {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2590}} Sep 13 15:52:30.193889 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] config_file = ['/etc/keystone/keystone.conf'] {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 13 15:52:30.194007 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] command line args: [] {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2591}} Sep 13 15:52:30.194089 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] config_source = [] {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 13 15:52:30.194189 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] debug = True {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 13 15:52:30.194272 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] config files: ['/etc/keystone/keystone.conf'] {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2592}} Sep 13 15:52:30.194394 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] default_log_levels = ['amqp=WARN', 'amqplib=WARN', 'boto=WARN', 'qpid=WARN', 'sqlalchemy=WARN', 'suds=INFO', 'oslo.messaging=INFO', 'oslo_messaging=INFO', 'iso8601=WARN', 'requests.packages.urllib3.connectionpool=WARN', 'urllib3.connectionpool=WARN', 'websocket=WARN', 'requests.packages.urllib3.util.retry=WARN', 'urllib3.util.retry=WARN', 'keystonemiddleware=WARN', 'routes.middleware=WARN', 'stevedore=WARN', 'taskflow=WARN', 'keystoneauth=WARN', 'oslo.cache=INFO', 'oslo_policy=INFO', 'dogpile.core.dogpile=INFO', 'dogpile=INFO', 'routes=INFO'] {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 13 15:52:30.194655 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] ================================================================================ {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2594}} Sep 13 15:52:30.194724 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] admin_token = **** {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 13 15:52:30.194792 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] default_publisher_id = None {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 13 15:52:30.194960 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] config_dir = [] {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 13 15:52:30.195041 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] insecure_debug = False {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 13 15:52:30.195151 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] config_file = ['/etc/keystone/keystone.conf'] {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 13 15:52:30.195248 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] instance_format = [instance: %(uuid)s] {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 13 15:52:30.195347 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] config_source = [] {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 13 15:52:30.195449 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] instance_uuid_format = [instance: %(uuid)s] {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 13 15:52:30.195551 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] debug = True {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 13 15:52:30.195652 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] list_limit = None {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 13 15:52:30.195747 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] default_log_levels = ['amqp=WARN', 'amqplib=WARN', 'boto=WARN', 'qpid=WARN', 'sqlalchemy=WARN', 'suds=INFO', 'oslo.messaging=INFO', 'oslo_messaging=INFO', 'iso8601=WARN', 'requests.packages.urllib3.connectionpool=WARN', 'urllib3.connectionpool=WARN', 'websocket=WARN', 'requests.packages.urllib3.util.retry=WARN', 'urllib3.util.retry=WARN', 'keystonemiddleware=WARN', 'routes.middleware=WARN', 'stevedore=WARN', 'taskflow=WARN', 'keystoneauth=WARN', 'oslo.cache=INFO', 'oslo_policy=INFO', 'dogpile.core.dogpile=INFO', 'dogpile=INFO', 'routes=INFO'] {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 13 15:52:30.195878 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] log_config_append = None {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 13 15:52:30.195983 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] default_publisher_id = None {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 13 15:52:30.196078 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] log_date_format = %Y-%m-%d %H:%M:%S {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 13 15:52:30.196182 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] insecure_debug = False {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 13 15:52:30.196281 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] log_dir = None {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 13 15:52:30.196374 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] instance_format = [instance: %(uuid)s] {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 13 15:52:30.196473 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] log_file = None {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 13 15:52:30.196577 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] log_rotate_interval = 1 {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 13 15:52:30.196676 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] instance_uuid_format = [instance: %(uuid)s] {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 13 15:52:30.196800 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] list_limit = None {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 13 15:52:30.196875 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] log_rotate_interval_type = days {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 13 15:52:30.197103 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] log_config_append = None {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 13 15:52:30.197696 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] log_date_format = %Y-%m-%d %H:%M:%S {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 13 15:52:30.197788 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] log_rotation_type = none {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 13 15:52:30.197883 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] log_dir = None {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 13 15:52:30.198052 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] logging_context_format_string = %(color)s%(levelname)s %(name)s [%(global_request_id)s %(request_id)s %(project_name)s %(user_name)s%(color)s] %(instance)s%(color)s%(message)s {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 13 15:52:30.198158 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] log_file = None {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 13 15:52:30.198237 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] logging_debug_format_suffix = {{(pid=%(process)d) %(funcName)s %(pathname)s:%(lineno)d}} {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 13 15:52:30.198364 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] log_rotate_interval = 1 {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 13 15:52:30.198447 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] logging_default_format_string = %(color)s%(levelname)s %(name)s [-%(color)s] %(instance)s%(color)s%(message)s {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 13 15:52:30.198525 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] log_rotate_interval_type = days {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 13 15:52:30.198644 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] logging_exception_prefix = ERROR %(name)s %(instance)s {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 13 15:52:30.198766 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] log_rotation_type = none {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 13 15:52:30.198878 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] logging_user_identity_format = %(user)s %(project)s %(domain)s %(system_scope)s %(user_domain)s %(project_domain)s {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 13 15:52:30.199043 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] logging_context_format_string = %(color)s%(levelname)s %(name)s [%(global_request_id)s %(request_id)s %(project_name)s %(user_name)s%(color)s] %(instance)s%(color)s%(message)s {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 13 15:52:30.199132 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] max_logfile_count = 30 {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 13 15:52:30.199210 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] logging_debug_format_suffix = {{(pid=%(process)d) %(funcName)s %(pathname)s:%(lineno)d}} {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 13 15:52:30.199364 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] max_logfile_size_mb = 200 {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 13 15:52:30.199447 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] logging_default_format_string = %(color)s%(levelname)s %(name)s [-%(color)s] %(instance)s%(color)s%(message)s {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 13 15:52:30.199603 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] max_param_size = 64 {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 13 15:52:30.199693 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] logging_exception_prefix = ERROR %(name)s %(instance)s {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 13 15:52:30.199842 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] max_project_tree_depth = 5 {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 13 15:52:30.199963 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] logging_user_identity_format = %(user)s %(project)s %(domain)s %(system_scope)s %(user_domain)s %(project_domain)s {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 13 15:52:30.200095 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] max_logfile_count = 30 {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 13 15:52:30.200177 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] max_token_size = 16384 {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 13 15:52:30.200332 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] max_logfile_size_mb = 200 {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 13 15:52:30.200420 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] notification_format = cadf {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 13 15:52:30.200571 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] max_param_size = 64 {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 13 15:52:30.200659 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] notification_opt_out = ['identity.authenticate.success', 'identity.authenticate.pending', 'identity.authenticate.failed'] {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 13 15:52:30.200808 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] max_project_tree_depth = 5 {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 13 15:52:30.201063 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] max_token_size = 16384 {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 13 15:52:30.201146 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] public_endpoint = http://10.222.0.50/identity {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 13 15:52:30.201260 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] publish_errors = False {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 13 15:52:30.201342 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] notification_format = cadf {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 13 15:52:30.201497 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] pydev_debug_host = None {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 13 15:52:30.201623 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] notification_opt_out = ['identity.authenticate.success', 'identity.authenticate.pending', 'identity.authenticate.failed'] {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 13 15:52:30.201734 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] pydev_debug_port = None {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 13 15:52:30.201922 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] rate_limit_burst = 0 {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 13 15:52:30.202045 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] public_endpoint = http://10.222.0.50/identity {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 13 15:52:30.202212 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] rate_limit_except_level = CRITICAL {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 13 15:52:30.202311 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] publish_errors = False {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 13 15:52:30.202401 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] rate_limit_interval = 0 {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 13 15:52:30.202465 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] pydev_debug_host = None {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 13 15:52:30.202633 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] standard_threads = False {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 13 15:52:30.202740 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] pydev_debug_port = None {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 13 15:52:30.202813 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] rate_limit_burst = 0 {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 13 15:52:30.202879 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] strict_password_check = False {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 13 15:52:30.203090 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] rate_limit_except_level = CRITICAL {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 13 15:52:30.203177 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] syslog_log_facility = LOG_USER {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 13 15:52:30.203319 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] use_eventlog = False {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 13 15:52:30.203408 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] rate_limit_interval = 0 {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 13 15:52:30.203472 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] use_journal = False {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 13 15:52:30.203561 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] standard_threads = False {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 13 15:52:30.203691 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] use_json = False {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 13 15:52:30.203800 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] strict_password_check = False {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 13 15:52:30.203880 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] use_stderr = False {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 13 15:52:30.204001 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] use_syslog = False {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 13 15:52:30.204080 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] syslog_log_facility = LOG_USER {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 13 15:52:30.204197 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] watch_log_file = False {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 13 15:52:30.204281 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] use_eventlog = False {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 13 15:52:30.204380 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] use_journal = False {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 13 15:52:30.204457 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] oslo_concurrency.disable_process_locking = False {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.204561 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] use_json = False {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 13 15:52:30.204685 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] oslo_concurrency.lock_path = None {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.204768 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] use_stderr = False {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 13 15:52:30.204905 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] use_syslog = False {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 13 15:52:30.204981 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] profiler.connection_string = messaging:// {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.205141 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] watch_log_file = False {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 13 15:52:30.205230 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] profiler.enabled = False {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.205402 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] oslo_concurrency.disable_process_locking = False {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.205480 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] profiler.es_doc_type = notification {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.205644 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] oslo_concurrency.lock_path = None {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.205721 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] profiler.es_scroll_size = 10000 {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.205893 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] profiler.es_scroll_time = 2m {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.205982 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] profiler.connection_string = messaging:// {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.206135 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] profiler.filter_error_trace = False {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.206214 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] profiler.enabled = False {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.206383 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] profiler.es_doc_type = notification {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.206459 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] profiler.hmac_keys = SECRET_KEY {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.206625 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] profiler.es_scroll_size = 10000 {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.206713 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] profiler.sentinel_service_name = mymaster {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.206873 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] profiler.socket_timeout = 0.1 {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.206962 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] profiler.es_scroll_time = 2m {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.207134 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] profiler.trace_sqlalchemy = False {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.207220 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] profiler.filter_error_trace = False {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.207383 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] profiler.hmac_keys = SECRET_KEY {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.207468 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_buffer_size = 1000 {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.207619 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_enabled = False {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.207702 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] profiler.sentinel_service_name = mymaster {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.207869 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_process_name = {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.207955 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] profiler.socket_timeout = 0.1 {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.208104 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] profiler.trace_sqlalchemy = False {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.208189 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_socket_file = /var/tmp/metrics_collector.sock {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.208344 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_thread_stop_timeout = 10 {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.208422 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_buffer_size = 1000 {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.208580 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_enabled = False {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.208662 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] oslo_middleware.http_basic_auth_user_file = /etc/htpasswd {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.208829 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_process_name = {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.208915 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] audit.namespace = openstack {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.209080 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_socket_file = /var/tmp/metrics_collector.sock {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.209164 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] oslo_policy.enforce_new_defaults = False {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.209317 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] oslo_policy.enforce_scope = False {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.209393 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_thread_stop_timeout = 10 {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.209573 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] oslo_policy.policy_default_rule = default {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.209722 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] oslo_middleware.http_basic_auth_user_file = /etc/htpasswd {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.209830 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] audit.namespace = openstack {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.209920 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] oslo_policy.policy_dirs = ['policy.d'] {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.210080 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] oslo_policy.enforce_new_defaults = False {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.210164 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] oslo_policy.policy_file = policy.yaml {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.210323 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] oslo_policy.enforce_scope = False {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.210408 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] oslo_policy.remote_content_type = application/x-www-form-urlencoded {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.210555 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] oslo_policy.remote_ssl_ca_crt_file = None {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.210631 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] oslo_policy.policy_default_rule = default {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.210790 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] oslo_policy.remote_ssl_client_crt_file = None {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.210874 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] oslo_policy.policy_dirs = ['policy.d'] {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.211047 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] oslo_policy.remote_ssl_client_key_file = None {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.211130 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] oslo_policy.policy_file = policy.yaml {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.211286 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] oslo_policy.remote_ssl_verify_server_crt = False {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.211366 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] oslo_policy.remote_content_type = application/x-www-form-urlencoded {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.211537 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] keystone_authtoken.admin_password = **** {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.211620 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] oslo_policy.remote_ssl_ca_crt_file = None {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.211781 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] oslo_policy.remote_ssl_client_crt_file = None {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.211859 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] keystone_authtoken.admin_tenant_name = admin {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.212013 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] keystone_authtoken.admin_token = **** {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.212100 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] oslo_policy.remote_ssl_client_key_file = None {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.212246 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] keystone_authtoken.admin_user = None {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.212325 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] oslo_policy.remote_ssl_verify_server_crt = False {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.212538 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] keystone_authtoken.admin_password = **** {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.212647 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] keystone_authtoken.auth_admin_prefix = {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.212823 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] keystone_authtoken.admin_tenant_name = admin {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.212905 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] keystone_authtoken.auth_host = 127.0.0.1 {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.213025 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] keystone_authtoken.admin_token = **** {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.213103 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] keystone_authtoken.auth_port = 35357 {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.213279 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] keystone_authtoken.auth_protocol = https {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.213363 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] keystone_authtoken.admin_user = None {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.213514 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] keystone_authtoken.auth_section = None {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.213593 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] keystone_authtoken.auth_admin_prefix = {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.213759 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] keystone_authtoken.auth_type = None {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.213842 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] keystone_authtoken.auth_host = 127.0.0.1 {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.213998 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] keystone_authtoken.auth_uri = None {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.214078 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] keystone_authtoken.auth_port = 35357 {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.214232 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] keystone_authtoken.auth_version = None {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.214310 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] keystone_authtoken.auth_protocol = https {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.214466 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] keystone_authtoken.cache = None {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.214544 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] keystone_authtoken.auth_section = None {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.214699 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] keystone_authtoken.cafile = None {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.214778 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] keystone_authtoken.auth_type = None {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.214951 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] keystone_authtoken.certfile = None {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.215033 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] keystone_authtoken.auth_uri = None {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.215192 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] keystone_authtoken.delay_auth_decision = False {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.215272 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] keystone_authtoken.auth_version = None {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.215415 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] keystone_authtoken.cache = None {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.215488 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] keystone_authtoken.enforce_token_bind = permissive {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.215651 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] keystone_authtoken.cafile = None {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.215750 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] keystone_authtoken.http_connect_timeout = None {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.215889 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] keystone_authtoken.certfile = None {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.215978 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] keystone_authtoken.http_request_max_retries = 3 {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.216118 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] keystone_authtoken.identity_uri = None {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.216197 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] keystone_authtoken.delay_auth_decision = False {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.216356 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] keystone_authtoken.include_service_catalog = True {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.216434 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] keystone_authtoken.enforce_token_bind = permissive {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.216589 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] keystone_authtoken.http_connect_timeout = None {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.216672 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] keystone_authtoken.insecure = False {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.216832 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] keystone_authtoken.http_request_max_retries = 3 {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.216942 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] keystone_authtoken.interface = internal {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.217064 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] keystone_authtoken.keyfile = None {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.217167 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] keystone_authtoken.identity_uri = None {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.217300 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] keystone_authtoken.include_service_catalog = True {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.217404 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_conn_get_timeout = 10 {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.217537 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_dead_retry = 300 {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.217648 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] keystone_authtoken.insecure = False {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.217779 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_maxsize = 10 {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.217899 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] keystone_authtoken.interface = internal {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.218012 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] keystone_authtoken.keyfile = None {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.218136 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_socket_timeout = 3 {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.218248 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_unused_timeout = 60 {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.218325 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_conn_get_timeout = 10 {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.218480 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] keystone_authtoken.memcache_secret_key = **** {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.218593 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_dead_retry = 300 {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.218720 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_maxsize = 10 {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.218806 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] keystone_authtoken.memcache_security_strategy = None {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.218976 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] keystone_authtoken.memcache_use_advanced_pool = True {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.219055 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_socket_timeout = 3 {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.219218 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_unused_timeout = 60 {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.219333 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] keystone_authtoken.memcached_servers = None {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.219450 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] keystone_authtoken.region_name = None {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.219539 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] keystone_authtoken.memcache_secret_key = **** {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.219699 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] keystone_authtoken.memcache_security_strategy = None {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.219779 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] keystone_authtoken.service_token_roles = ['service'] {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.219937 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] keystone_authtoken.service_token_roles_required = False {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.220017 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] keystone_authtoken.memcache_use_advanced_pool = True {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.220171 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] keystone_authtoken.service_type = None {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.220260 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] keystone_authtoken.memcached_servers = None {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.220405 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] keystone_authtoken.region_name = None {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.220497 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] keystone_authtoken.token_cache_time = 300 {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.220642 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] keystone_authtoken.www_authenticate_uri = None {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.220720 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] keystone_authtoken.service_token_roles = ['service'] {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.220893 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] keystone_authtoken.service_token_roles_required = False {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.220973 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] cache.backend = dogpile.cache.memcached {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.221130 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] keystone_authtoken.service_type = None {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.221218 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] cache.backend_argument = **** {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.221372 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] keystone_authtoken.token_cache_time = 300 {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.221492 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] cache.config_prefix = cache.oslo {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.221611 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] keystone_authtoken.www_authenticate_uri = None {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.221688 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] cache.dead_timeout = 60.0 {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.221846 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] cache.debug_cache_backend = False {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.221934 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] cache.backend = dogpile.cache.memcached {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.222070 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] cache.enable_retry_client = False {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.222143 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] cache.backend_argument = **** {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.222310 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] cache.enable_socket_keepalive = False {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.222397 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] cache.config_prefix = cache.oslo {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.222560 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] cache.enabled = True {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.222690 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] cache.dead_timeout = 60.0 {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.222801 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] cache.expiration_time = 600 {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.222880 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] cache.debug_cache_backend = False {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.223066 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] cache.enable_retry_client = False {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.223177 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] cache.hashclient_retry_attempts = 2 {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.223308 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] cache.enable_socket_keepalive = False {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.223396 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] cache.hashclient_retry_delay = 1.0 {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.223546 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] cache.memcache_dead_retry = 300 {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.223623 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] cache.enabled = True {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.223795 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] cache.expiration_time = 600 {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.223880 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] cache.memcache_password = {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.224035 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] cache.memcache_pool_connection_get_timeout = 10 {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.224126 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] cache.hashclient_retry_attempts = 2 {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.224274 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] cache.memcache_pool_flush_on_reconnect = False {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.224352 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] cache.hashclient_retry_delay = 1.0 {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.224512 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] cache.memcache_dead_retry = 300 {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.224588 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] cache.memcache_pool_maxsize = 10 {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.224746 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] cache.memcache_pool_unused_timeout = 60 {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.224830 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] cache.memcache_password = {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.224984 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] cache.memcache_sasl_enabled = False {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.225069 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] cache.memcache_pool_connection_get_timeout = 10 {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.225220 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] cache.memcache_pool_flush_on_reconnect = False {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.225298 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] cache.memcache_servers = ['localhost:11211'] {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.225461 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] cache.memcache_pool_maxsize = 10 {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.225546 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] cache.memcache_socket_timeout = 1.0 {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.225700 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] cache.memcache_pool_unused_timeout = 60 {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.225778 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] cache.memcache_username = {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.225940 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] cache.memcache_sasl_enabled = False {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.226023 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] cache.proxies = [] {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.226180 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] cache.retry_attempts = 2 {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.226268 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] cache.memcache_servers = ['localhost:11211'] {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.226429 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] cache.retry_delay = 0.0 {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.226507 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] cache.memcache_socket_timeout = 1.0 {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.226671 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] cache.socket_keepalive_count = 1 {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.226749 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] cache.memcache_username = {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.226938 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] cache.proxies = [] {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.227094 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] cache.socket_keepalive_idle = 1 {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.227171 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] cache.socket_keepalive_interval = 1 {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.227241 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] cache.retry_attempts = 2 {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.227393 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] cache.tls_allowed_ciphers = None {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.227472 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] cache.retry_delay = 0.0 {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.227629 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] cache.tls_cafile = None {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.227761 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] cache.socket_keepalive_count = 1 {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.227849 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] cache.tls_certfile = None {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.227922 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] cache.socket_keepalive_idle = 1 {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.228093 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] cache.socket_keepalive_interval = 1 {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.228676 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] cache.tls_enabled = False {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.228869 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] cache.tls_keyfile = None {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.229047 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] cache.tls_allowed_ciphers = None {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.229134 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] application_credential.cache_time = None {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.229328 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] cache.tls_cafile = None {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.229419 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] application_credential.caching = True {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.229536 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] cache.tls_certfile = None {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.229615 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] application_credential.driver = sql {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.229779 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] cache.tls_enabled = False {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.229866 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] application_credential.user_limit = -1 {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.230016 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] cache.tls_keyfile = None {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.230095 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] assignment.driver = sql {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.230263 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] application_credential.cache_time = None {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.230341 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] assignment.prohibited_implied_role = ['admin'] {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.230506 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] application_credential.caching = True {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.230593 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] auth.application_credential = None {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.230755 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] auth.external = None {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.230840 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] application_credential.driver = sql {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.231014 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] auth.mapped = None {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.231112 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] application_credential.user_limit = -1 {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.231275 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] assignment.driver = sql {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.231363 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] auth.methods = ['external', 'password', 'token', 'oauth1', 'mapped', 'application_credential'] {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.231535 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] auth.oauth1 = None {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.231621 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] assignment.prohibited_implied_role = ['admin'] {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.231770 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] auth.password = None {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.231849 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] auth.application_credential = None {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.232004 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] auth.external = None {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.232082 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] auth.token = None {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.232241 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] auth.mapped = None {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.232330 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] catalog.cache_time = None {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.232478 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] catalog.caching = True {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.232556 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] auth.methods = ['external', 'password', 'token', 'oauth1', 'mapped', 'application_credential'] {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.232730 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] catalog.driver = sql {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.232808 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] auth.oauth1 = None {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.232965 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] catalog.list_limit = None {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.233052 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] auth.password = None {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.233211 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] auth.token = None {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.233290 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] catalog.template_file = default_catalog.templates {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.233459 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] credential.auth_ttl = 15 {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.233537 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] catalog.cache_time = None {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.233693 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] credential.cache_time = None {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.233779 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] catalog.caching = True {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.233933 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] credential.caching = True {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.234012 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] catalog.driver = sql {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.234169 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] catalog.list_limit = None {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.234246 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] credential.driver = sql {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.234422 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] catalog.template_file = default_catalog.templates {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.234510 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] credential.key_repository = /etc/keystone/credential-keys/ {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.234672 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] credential.provider = fernet {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.234750 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] credential.auth_ttl = 15 {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.234955 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] credential.cache_time = None {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.235037 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] credential.user_limit = -1 {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.235172 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] credential.caching = True {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.235263 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] domain_config.cache_time = 300 {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.235406 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] domain_config.caching = True {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.235486 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] credential.driver = sql {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.235655 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] domain_config.driver = sql {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.235778 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] credential.key_repository = /etc/keystone/credential-keys/ {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.235904 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] credential.provider = fernet {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.235990 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] endpoint_filter.driver = sql {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.236141 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] endpoint_filter.return_all_endpoints_if_no_filter = True {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.236221 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] credential.user_limit = -1 {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.236386 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] domain_config.cache_time = 300 {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.236464 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] endpoint_policy.driver = sql {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.236624 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] domain_config.caching = True {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.236711 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] eventlet_server.admin_bind_host = 0.0.0.0 {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.236875 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] domain_config.driver = sql {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.236955 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] eventlet_server.admin_port = 35357 {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.237132 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] endpoint_filter.driver = sql {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.237211 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] eventlet_server.public_bind_host = 0.0.0.0 {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.237373 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] endpoint_filter.return_all_endpoints_if_no_filter = True {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.237459 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] eventlet_server.public_port = 5000 {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.237628 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] endpoint_policy.driver = sql {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.237708 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] federation.assertion_prefix = {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.237863 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] federation.caching = True {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.237941 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] eventlet_server.admin_bind_host = 0.0.0.0 {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.238104 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] federation.default_authorization_ttl = 0 {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.238191 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] eventlet_server.admin_port = 35357 {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.238352 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] federation.driver = sql {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.238440 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] eventlet_server.public_bind_host = 0.0.0.0 {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.238602 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] federation.federated_domain_name = Federated {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.238682 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] eventlet_server.public_port = 5000 {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.238835 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] federation.remote_id_attribute = None {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.238940 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] federation.assertion_prefix = {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.239111 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] federation.sso_callback_template = /etc/keystone/sso_callback_template.html {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.239193 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] federation.caching = True {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.239351 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] federation.trusted_dashboard = [] {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.239430 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] federation.default_authorization_ttl = 0 {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.239597 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] federation.driver = sql {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.239683 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] fernet_receipts.key_repository = /etc/keystone/fernet-keys/ {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.239835 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] fernet_receipts.max_active_keys = 3 {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.239914 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] federation.federated_domain_name = Federated {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.240065 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] federation.remote_id_attribute = None {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.240143 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] fernet_tokens.key_repository = /etc/keystone/fernet-keys/ {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.240315 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] federation.sso_callback_template = /etc/keystone/sso_callback_template.html {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.240396 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] fernet_tokens.max_active_keys = 3 {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.240556 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] federation.trusted_dashboard = [] {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.240635 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] identity.cache_time = 600 {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.240792 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] identity.caching = True {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.240869 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] fernet_receipts.key_repository = /etc/keystone/fernet-keys/ {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.241037 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] fernet_receipts.max_active_keys = 3 {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.241116 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] identity.default_domain_id = default {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.241284 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] identity.domain_config_dir = /etc/keystone/domains {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.241362 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] fernet_tokens.key_repository = /etc/keystone/fernet-keys/ {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.241521 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] identity.domain_configurations_from_database = False {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.241600 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] fernet_tokens.max_active_keys = 3 {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.241744 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] identity.domain_specific_drivers_enabled = False {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.241821 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] identity.cache_time = 600 {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.241992 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] identity.driver = sql {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.242071 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] identity.caching = True {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.242231 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] identity.list_limit = None {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.242346 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] identity.default_domain_id = default {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.242476 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] identity.max_password_length = 4096 {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.242557 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] identity.domain_config_dir = /etc/keystone/domains {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.242709 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] identity.domain_configurations_from_database = False {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.242788 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] identity.password_hash_algorithm = bcrypt {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.242978 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] identity.domain_specific_drivers_enabled = False {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.243062 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] identity.password_hash_rounds = 4 {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.243212 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] identity.salt_bytesize = None {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.243293 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] identity.driver = sql {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.243446 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] identity.scrypt_block_size = None {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.243525 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] identity.list_limit = None {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.243679 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] identity.scrypt_parallelism = None {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.243811 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] identity.max_password_length = 4096 {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.244572 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] identity.password_hash_algorithm = bcrypt {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.244658 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] identity_mapping.backward_compatible_ids = True {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.244731 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] identity.password_hash_rounds = 4 {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.244802 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] identity_mapping.driver = sql {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.244964 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] identity.salt_bytesize = None {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.245044 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] identity_mapping.generator = sha256 {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.245201 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] identity.scrypt_block_size = None {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.245283 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] jwt_tokens.jws_private_key_repository = /etc/keystone/jws-keys/private {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.245436 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] identity.scrypt_parallelism = None {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.245516 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] jwt_tokens.jws_public_key_repository = /etc/keystone/jws-keys/public {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.245683 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] identity_mapping.backward_compatible_ids = True {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.245762 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] ldap.alias_dereferencing = default {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.245934 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] identity_mapping.driver = sql {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.246021 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] ldap.auth_pool_connection_lifetime = 60 {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.246185 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] identity_mapping.generator = sha256 {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.246266 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] ldap.auth_pool_size = 100 {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.246416 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] ldap.chase_referrals = None {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.246493 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] jwt_tokens.jws_private_key_repository = /etc/keystone/jws-keys/private {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.246656 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] ldap.connection_timeout = -1 {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.246737 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] jwt_tokens.jws_public_key_repository = /etc/keystone/jws-keys/public {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.246891 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] ldap.debug_level = None {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.246984 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] ldap.alias_dereferencing = default {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.247154 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] ldap.group_ad_nesting = False {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.247282 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] ldap.auth_pool_connection_lifetime = 60 {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.247400 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] ldap.group_additional_attribute_mapping = [] {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.247482 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] ldap.auth_pool_size = 100 {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.247643 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] ldap.group_attribute_ignore = [] {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.247722 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] ldap.chase_referrals = None {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.247881 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] ldap.connection_timeout = -1 {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.247959 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] ldap.group_desc_attribute = description {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.248114 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] ldap.group_filter = None {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.248195 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] ldap.debug_level = None {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.248352 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] ldap.group_ad_nesting = False {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.248430 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] ldap.group_id_attribute = cn {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.248597 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] ldap.group_additional_attribute_mapping = [] {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.248676 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] ldap.group_member_attribute = member {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.248833 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] ldap.group_members_are_ids = False {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.248912 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] ldap.group_attribute_ignore = [] {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.249082 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] ldap.group_desc_attribute = description {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.249207 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] ldap.group_name_attribute = ou {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.249317 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] ldap.group_filter = None {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.249397 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] ldap.group_objectclass = groupOfNames {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.249549 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] ldap.group_tree_dn = None {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.249629 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] ldap.group_id_attribute = cn {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.249790 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] ldap.page_size = 0 {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.249869 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] ldap.group_member_attribute = member {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.250024 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] ldap.password = **** {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.250110 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] ldap.group_members_are_ids = False {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.250263 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] ldap.pool_connection_lifetime = 600 {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.250342 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] ldap.group_name_attribute = ou {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.250501 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] ldap.pool_connection_timeout = -1 {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.250579 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] ldap.group_objectclass = groupOfNames {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.250732 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] ldap.group_tree_dn = None {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.250818 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] ldap.pool_retry_delay = 0.1 {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.250994 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] ldap.pool_retry_max = 3 {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.251077 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] ldap.page_size = 0 {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.251229 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] ldap.password = **** {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.251348 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] ldap.pool_size = 10 {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.251471 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] ldap.pool_connection_lifetime = 600 {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.251605 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] ldap.query_scope = one {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.251711 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] ldap.pool_connection_timeout = -1 {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.251858 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] ldap.suffix = cn=example,cn=com {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.251956 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] ldap.pool_retry_delay = 0.1 {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.252091 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] ldap.tls_cacertdir = None {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.252197 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] ldap.pool_retry_max = 3 {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.252325 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] ldap.tls_cacertfile = None {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.252548 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] ldap.pool_size = 10 {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.252634 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] ldap.tls_req_cert = demand {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.252807 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] ldap.query_scope = one {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.252886 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] ldap.url = ldap://localhost {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.253046 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] ldap.use_auth_pool = True {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.253131 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] ldap.suffix = cn=example,cn=com {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.253288 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] ldap.tls_cacertdir = None {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.253367 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] ldap.use_pool = True {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.253525 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] ldap.use_tls = False {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.253604 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] ldap.tls_cacertfile = None {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.253760 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] ldap.user = None {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.253847 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] ldap.tls_req_cert = demand {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.254004 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] ldap.user_additional_attribute_mapping = [] {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.254083 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] ldap.url = ldap://localhost {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.254251 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] ldap.use_auth_pool = True {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.254339 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] ldap.user_attribute_ignore = ['default_project_id'] {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.254488 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] ldap.use_pool = True {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.254576 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] ldap.user_default_project_id_attribute = None {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.254729 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] ldap.use_tls = False {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.254809 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] ldap.user_description_attribute = description {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.254988 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] ldap.user = None {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.255069 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] ldap.user_enabled_attribute = enabled {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.255235 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] ldap.user_additional_attribute_mapping = [] {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.255326 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] ldap.user_enabled_default = True {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.255473 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] ldap.user_enabled_emulation = False {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.255552 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] ldap.user_attribute_ignore = ['default_project_id'] {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.255708 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] ldap.user_enabled_emulation_dn = None {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.255786 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] ldap.user_default_project_id_attribute = None {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.255948 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] ldap.user_enabled_emulation_use_group_config = False {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.256036 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] ldap.user_description_attribute = description {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.256186 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] ldap.user_enabled_invert = False {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.256266 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] ldap.user_enabled_attribute = enabled {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.256428 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] ldap.user_enabled_mask = 0 {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.256507 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] ldap.user_enabled_default = True {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.256661 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] ldap.user_filter = None {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.256749 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] ldap.user_enabled_emulation = False {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.256893 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] ldap.user_enabled_emulation_dn = None {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.256971 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] ldap.user_id_attribute = cn {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.257134 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] ldap.user_enabled_emulation_use_group_config = False {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.257212 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] ldap.user_mail_attribute = mail {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.257373 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] ldap.user_enabled_invert = False {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.257460 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] ldap.user_name_attribute = sn {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.257614 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] ldap.user_enabled_mask = 0 {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.257700 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] ldap.user_objectclass = inetOrgPerson {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.257849 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] ldap.user_filter = None {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.257929 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] ldap.user_pass_attribute = userPassword {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.258077 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] ldap.user_tree_dn = None {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.258163 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] ldap.user_id_attribute = cn {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.258327 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] memcache.dead_retry = 300 {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.258406 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] ldap.user_mail_attribute = mail {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.258570 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] memcache.pool_connection_get_timeout = 10 {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.258649 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] ldap.user_name_attribute = sn {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.258813 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] memcache.pool_maxsize = 10 {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.258953 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] ldap.user_objectclass = inetOrgPerson {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.259076 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] memcache.pool_unused_timeout = 60 {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.259157 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] ldap.user_pass_attribute = userPassword {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.259311 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] ldap.user_tree_dn = None {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.259401 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] memcache.socket_timeout = 3 {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.259591 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] memcache.dead_retry = 300 {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.259682 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] oauth1.access_token_duration = 86400 {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.259835 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] memcache.pool_connection_get_timeout = 10 {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.259915 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] oauth1.driver = sql {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.260079 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] memcache.pool_maxsize = 10 {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.260157 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] oauth1.request_token_duration = 28800 {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.260320 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] memcache.pool_unused_timeout = 60 {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.260407 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] policy.driver = sql {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.260561 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] memcache.socket_timeout = 3 {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.260646 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] policy.list_limit = None {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.260811 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] oauth1.access_token_duration = 86400 {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.260891 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] receipt.cache_on_issue = True {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.261064 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] oauth1.driver = sql {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.261151 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] receipt.cache_time = 300 {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.261305 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] oauth1.request_token_duration = 28800 {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.261384 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] receipt.caching = True {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.261545 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] receipt.expiration = 300 {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.261622 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] policy.driver = sql {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.261784 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] policy.list_limit = None {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.261869 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] receipt.provider = fernet {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.262026 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] resource.admin_project_domain_name = None {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.262105 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] receipt.cache_on_issue = True {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.262262 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] resource.admin_project_name = None {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.262340 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] receipt.cache_time = 300 {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.262500 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] receipt.caching = True {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.262586 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] resource.cache_time = None {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.262742 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] resource.caching = True {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.262821 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] receipt.expiration = 300 {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.263018 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] resource.domain_name_url_safe = off {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.263100 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] receipt.provider = fernet {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.263264 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] resource.admin_project_domain_name = None {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.263355 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] resource.driver = sql {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.263501 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] resource.list_limit = None {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.263580 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] resource.admin_project_name = None {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.263741 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] resource.cache_time = None {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.263820 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] resource.project_name_url_safe = off {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.263986 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] resource.caching = True {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.264074 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] revoke.cache_time = 3600 {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.264223 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] revoke.caching = True {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.264301 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] resource.domain_name_url_safe = off {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.264477 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] revoke.driver = sql {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.264556 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] resource.driver = sql {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.264715 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] resource.list_limit = None {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.264801 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] revoke.expiration_buffer = 1800 {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.264954 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] role.cache_time = None {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.265033 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] resource.project_name_url_safe = off {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.265191 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] role.caching = True {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.265270 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] revoke.cache_time = 3600 {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.265431 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] revoke.caching = True {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.265587 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] role.driver = sql {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.265663 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] role.list_limit = None {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.265739 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] revoke.driver = sql {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.265896 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] saml.assertion_expiration_time = 3600 {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.265976 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] revoke.expiration_buffer = 1800 {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.266137 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] role.cache_time = None {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.266216 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] saml.certfile = /etc/keystone/ssl/certs/signing_cert.pem {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.266377 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] role.caching = True {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.266456 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] saml.idp_contact_company = Example, Inc. {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.266622 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] saml.idp_contact_email = support@example.com {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.266701 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] role.driver = sql {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.266854 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] role.list_limit = None {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.266973 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] saml.idp_contact_name = SAML Identity Provider Support {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.267131 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] saml.idp_contact_surname = Support {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.267209 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] saml.assertion_expiration_time = 3600 {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.267379 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] saml.certfile = /etc/keystone/ssl/certs/signing_cert.pem {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.267457 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] saml.idp_contact_telephone = +1 800 555 0100 {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.267626 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] saml.idp_contact_type = other {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.267706 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] saml.idp_contact_company = Example, Inc. {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.267860 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] saml.idp_entity_id = None {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.267962 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] saml.idp_contact_email = support@example.com {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.268110 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] saml.idp_contact_name = SAML Identity Provider Support {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.268188 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] saml.idp_lang = en {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.268355 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] saml.idp_metadata_path = /etc/keystone/saml2_idp_metadata.xml {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.268434 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] saml.idp_contact_surname = Support {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.268604 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] saml.idp_contact_telephone = +1 800 555 0100 {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.268682 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] saml.idp_organization_display_name = OpenStack SAML Identity Provider {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.268850 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] saml.idp_organization_name = SAML Identity Provider {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.268928 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] saml.idp_contact_type = other {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.269084 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] saml.idp_entity_id = None {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.269244 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] saml.idp_organization_url = https://example.com/ {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.269323 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] saml.idp_lang = en {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.269485 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] saml.idp_sso_endpoint = None {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.269564 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] saml.idp_metadata_path = /etc/keystone/saml2_idp_metadata.xml {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.269736 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] saml.keyfile = /etc/keystone/ssl/private/signing_key.pem {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.269823 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] saml.idp_organization_display_name = OpenStack SAML Identity Provider {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.269988 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] saml.relay_state_prefix = ss:mem: {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.270067 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] saml.idp_organization_name = SAML Identity Provider {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.270237 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] saml.xmlsec1_binary = xmlsec1 {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.270422 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] saml.idp_organization_url = https://example.com/ {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.270500 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] security_compliance.change_password_upon_first_use = False {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.270666 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] saml.idp_sso_endpoint = None {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.270746 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] security_compliance.disable_user_account_days_inactive = None {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.270937 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] saml.keyfile = /etc/keystone/ssl/private/signing_key.pem {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.271020 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] security_compliance.lockout_duration = 10 {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.271197 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] saml.relay_state_prefix = ss:mem: {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.271287 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] security_compliance.lockout_failure_attempts = 2 {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.271449 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] saml.xmlsec1_binary = xmlsec1 {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.271528 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] security_compliance.minimum_password_age = 0 {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.271683 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] security_compliance.password_expires_days = None {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.271761 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] security_compliance.change_password_upon_first_use = False {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.271918 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] security_compliance.password_regex = None {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.272009 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] security_compliance.disable_user_account_days_inactive = None {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.272154 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] security_compliance.password_regex_description = None {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.272234 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] security_compliance.lockout_duration = 10 {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.272409 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] security_compliance.unique_last_password_count = 2 {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.272487 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] security_compliance.lockout_failure_attempts = 2 {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.272650 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] security_compliance.minimum_password_age = 0 {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.272735 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] shadow_users.driver = sql {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.272886 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] security_compliance.password_expires_days = None {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.272965 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] token.allow_expired_window = 172800 {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.273121 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] security_compliance.password_regex = None {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.273200 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] token.allow_rescope_scoped_token = True {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.273356 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] token.cache_on_issue = True {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.273442 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] security_compliance.password_regex_description = None {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.273591 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] token.cache_time = None {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.273670 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] security_compliance.unique_last_password_count = 2 {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.273830 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] token.caching = True {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.273908 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] shadow_users.driver = sql {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.274070 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] token.expiration = 3600 {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.274156 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] token.allow_expired_window = 172800 {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.274319 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] token.allow_rescope_scoped_token = True {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.274397 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] token.provider = fernet {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.274560 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] token.revoke_by_id = True {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.274637 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] token.cache_on_issue = True {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.274795 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] token.cache_time = None {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.274881 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] tokenless_auth.issuer_attribute = SSL_CLIENT_I_DN {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.275107 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] token.caching = True {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.275191 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] tokenless_auth.protocol = x509 {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.275355 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] token.expiration = 3600 {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.275547 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] tokenless_auth.trusted_issuer = [] {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.275669 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] token.provider = fernet {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.275805 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] totp.included_previous_windows = 1 {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.275933 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] token.revoke_by_id = True {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.276060 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] trust.allow_redelegation = False {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.276197 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] tokenless_auth.issuer_attribute = SSL_CLIENT_I_DN {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.276317 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] trust.driver = sql {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.276450 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] tokenless_auth.protocol = x509 {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.276562 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] trust.max_redelegation_count = 3 {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.276694 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] tokenless_auth.trusted_issuer = [] {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.276807 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] unified_limit.cache_time = None {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.276944 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] totp.included_previous_windows = 1 {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.277046 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] unified_limit.caching = True {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.277193 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] trust.allow_redelegation = False {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.277280 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] unified_limit.driver = sql {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.277445 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] trust.driver = sql {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.277562 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] unified_limit.enforcement_model = flat {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.277686 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] trust.max_redelegation_count = 3 {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.277797 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] unified_limit.list_limit = None {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.277929 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] unified_limit.cache_time = None {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.278044 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] wsgi.debug_middleware = False {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.278170 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] unified_limit.caching = True {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.278311 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] database.backend = sqlalchemy {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.278421 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] unified_limit.driver = sql {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.278573 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] database.connection = **** {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.278698 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] unified_limit.enforcement_model = flat {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.278821 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] database.connection_debug = 0 {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.278965 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] unified_limit.list_limit = None {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.279094 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] database.connection_parameters = {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.279215 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] wsgi.debug_middleware = False {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.279336 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] database.connection_recycle_time = 3600 {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.279483 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] database.backend = sqlalchemy {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.279562 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] database.connection_trace = False {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.279745 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] database.connection = **** {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.279825 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] database.db_inc_retry_interval = True {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.279993 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] database.connection_debug = 0 {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.280074 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] database.db_max_retries = 20 {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.280244 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] database.connection_parameters = {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.280323 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] database.db_max_retry_interval = 10 {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.280479 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] database.db_retry_interval = 1 {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.280557 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] database.connection_recycle_time = 3600 {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.280721 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] database.connection_trace = False {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.280807 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] database.max_overflow = 50 {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.280957 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] database.max_pool_size = 5 {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.281035 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] database.db_inc_retry_interval = True {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.281194 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] database.db_max_retries = 20 {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.281272 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] database.max_retries = 10 {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.281429 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] database.mysql_enable_ndb = False {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.281509 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] database.db_max_retry_interval = 10 {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.281666 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] database.db_retry_interval = 1 {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.281743 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] database.mysql_sql_mode = TRADITIONAL {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.281902 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] database.pool_timeout = None {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.281980 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] database.max_overflow = 50 {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.282139 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] database.max_pool_size = 5 {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.282218 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] database.retry_interval = 10 {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.282371 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] database.slave_connection = **** {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.282448 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] database.max_retries = 10 {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.282610 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] database.mysql_enable_ndb = False {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.282695 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] database.sqlite_synchronous = True {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.282846 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] database.use_db_reconnect = False {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.282970 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] database.mysql_sql_mode = TRADITIONAL {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.283047 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG uwsgi [-] ******************************************************************************** {{(pid=81438) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2613}} Sep 13 15:52:30.283183 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] database.pool_timeout = None {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.283432 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] database.retry_interval = 10 {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.283667 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] database.slave_connection = **** {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.283911 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] database.sqlite_synchronous = True {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.284150 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] database.use_db_reconnect = False {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:52:30.284321 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG uwsgi [-] ******************************************************************************** {{(pid=81437) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2613}} Sep 13 15:52:30.313708 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_user_delete_callback` subscribed to event `identity.user.deleted`. {{(pid=81438) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 13 15:52:30.313885 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_user_delete_callback` subscribed to event `identity.user.disabled`. {{(pid=81438) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 13 15:52:30.314109 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_assignment_removal` subscribed to event `identity.remove_application_credentials_for_user.internal`. {{(pid=81438) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 13 15:52:30.314760 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_user_delete_callback` subscribed to event `identity.user.deleted`. {{(pid=81437) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 13 15:52:30.315034 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_user_delete_callback` subscribed to event `identity.user.disabled`. {{(pid=81437) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 13 15:52:30.315268 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_assignment_removal` subscribed to event `identity.remove_application_credentials_for_user.internal`. {{(pid=81437) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 13 15:52:30.322419 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [-] Callback: `keystone.assignment.core.Manager._delete_domain_assignments` subscribed to event `identity.domain.deleted`. {{(pid=81438) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 13 15:52:30.323836 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [-] Callback: `keystone.assignment.core.Manager._delete_domain_assignments` subscribed to event `identity.domain.deleted`. {{(pid=81437) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 13 15:52:30.350145 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [-] Callback: `keystone.catalog.core.Manager._on_project_or_endpoint_delete` subscribed to event `identity.project.deleted`. {{(pid=81438) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 13 15:52:30.350380 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [-] Callback: `keystone.catalog.core.Manager._on_project_or_endpoint_delete` subscribed to event `identity.endpoint.deleted`. {{(pid=81438) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 13 15:52:30.351540 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [-] Callback: `keystone.catalog.core.Manager._on_project_or_endpoint_delete` subscribed to event `identity.project.deleted`. {{(pid=81437) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 13 15:52:30.351777 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [-] Callback: `keystone.catalog.core.Manager._on_project_or_endpoint_delete` subscribed to event `identity.endpoint.deleted`. {{(pid=81437) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 13 15:52:30.404563 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [-] Callback: `keystone.federation.core.Manager._cleanup_identity_provider` subscribed to event `identity.domain_deleted.internal`. {{(pid=81438) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 13 15:52:30.405048 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [-] Callback: `keystone.federation.core.Manager._cleanup_identity_provider` subscribed to event `identity.domain_deleted.internal`. {{(pid=81437) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 13 15:52:30.453755 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [-] Callback: `keystone.identity.core.Manager._domain_deleted` subscribed to event `identity.domain_deleted.internal`. {{(pid=81438) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 13 15:52:30.453993 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [-] Callback: `keystone.identity.core.Manager._unset_default_project` subscribed to event `identity.project.deleted`. {{(pid=81438) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 13 15:52:30.454513 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [-] Callback: `keystone.identity.core.Manager._domain_deleted` subscribed to event `identity.domain_deleted.internal`. {{(pid=81437) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 13 15:52:30.454755 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [-] Callback: `keystone.identity.core.Manager._unset_default_project` subscribed to event `identity.project.deleted`. {{(pid=81437) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 13 15:52:30.611141 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._trust_callback` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=81438) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 13 15:52:30.611370 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._consumer_callback` subscribed to event `identity.OS-OAUTH1:consumer.deleted`. {{(pid=81438) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 13 15:52:30.611453 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._trust_callback` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=81437) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 13 15:52:30.611596 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.user.deleted`. {{(pid=81438) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 13 15:52:30.611681 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._consumer_callback` subscribed to event `identity.OS-OAUTH1:consumer.deleted`. {{(pid=81437) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 13 15:52:30.611815 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._project_callback` subscribed to event `identity.project.deleted`. {{(pid=81438) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 13 15:52:30.611901 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.user.deleted`. {{(pid=81437) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 13 15:52:30.612034 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.user.disabled`. {{(pid=81438) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 13 15:52:30.612116 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._project_callback` subscribed to event `identity.project.deleted`. {{(pid=81437) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 13 15:52:30.612251 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.persist_revocation_event_for_user.internal`. {{(pid=81438) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 13 15:52:30.612344 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.user.disabled`. {{(pid=81437) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 13 15:52:30.612492 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.persist_revocation_event_for_user.internal`. {{(pid=81437) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 13 15:52:30.635269 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=81438) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 13 15:52:30.635499 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.user.deleted`. {{(pid=81438) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 13 15:52:30.635591 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=81437) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 13 15:52:30.635730 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.domain.deleted`. {{(pid=81438) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 13 15:52:30.635856 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.user.deleted`. {{(pid=81437) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 13 15:52:30.635936 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.user.disabled`. {{(pid=81438) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 13 15:52:30.636011 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.domain.deleted`. {{(pid=81437) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 13 15:52:30.636149 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.domain.disabled`. {{(pid=81438) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 13 15:52:30.636233 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.user.disabled`. {{(pid=81437) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 13 15:52:30.636363 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.project.disabled`. {{(pid=81438) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 13 15:52:30.636447 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.domain.disabled`. {{(pid=81437) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 13 15:52:30.636577 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.invalidate_token_cache.internal`. {{(pid=81438) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 13 15:52:30.636658 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.project.disabled`. {{(pid=81437) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 13 15:52:30.636792 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.invalidate_token_cache.internal`. {{(pid=81437) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 13 15:52:30.647671 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [-] Callback: `keystone.trust.core.Manager._on_user_delete` subscribed to event `identity.user.deleted`. {{(pid=81438) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 13 15:52:30.647935 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [-] Callback: `keystone.trust.core.Manager._on_user_delete` subscribed to event `identity.user.deleted`. {{(pid=81437) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 13 15:52:30.661071 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=81438) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 13 15:52:30.661317 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.user.deleted`. {{(pid=81438) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 13 15:52:30.661536 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.domain.deleted`. {{(pid=81438) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 13 15:52:30.661672 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=81437) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 13 15:52:30.661764 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.user.disabled`. {{(pid=81438) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 13 15:52:30.661909 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.user.deleted`. {{(pid=81437) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 13 15:52:30.661996 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.domain.disabled`. {{(pid=81438) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 13 15:52:30.662135 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.domain.deleted`. {{(pid=81437) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 13 15:52:30.662220 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.project.disabled`. {{(pid=81438) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 13 15:52:30.662356 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.invalidate_token_cache.internal`. {{(pid=81438) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 13 15:52:30.662442 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.user.disabled`. {{(pid=81437) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 13 15:52:30.662572 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.domain.disabled`. {{(pid=81437) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 13 15:52:30.662788 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.project.disabled`. {{(pid=81437) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 13 15:52:30.663039 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.invalidate_token_cache.internal`. {{(pid=81437) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 13 15:52:30.679839 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/projects' {}] {{(pid=81438) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:52:30.680005 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/projects' {}] {{(pid=81437) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:52:30.680093 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.common [-] Adding additional resource route (alternate) to API auth: ['/OS-FEDERATION/projects' {}] {{(pid=81438) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:433}} Sep 13 15:52:30.680220 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.common [-] Adding additional resource route (alternate) to API auth: ['/OS-FEDERATION/projects' {}] {{(pid=81437) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:433}} Sep 13 15:52:30.680380 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/domains' {}] {{(pid=81438) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:52:30.680566 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/domains' {}] {{(pid=81437) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:52:30.680665 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.common [-] Adding additional resource route (alternate) to API auth: ['/OS-FEDERATION/domains' {}] {{(pid=81438) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:433}} Sep 13 15:52:30.680770 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.common [-] Adding additional resource route (alternate) to API auth: ['/OS-FEDERATION/domains' {}] {{(pid=81437) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:433}} Sep 13 15:52:30.680858 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/system' {}] {{(pid=81438) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:52:30.681047 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/system' {}] {{(pid=81437) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:52:30.681131 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/catalog' {}] {{(pid=81438) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:52:30.681354 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/catalog' {}] {{(pid=81437) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:52:30.681441 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/tokens/OS-PKI/revoked' {}] {{(pid=81438) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:52:30.681716 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/tokens/OS-PKI/revoked' {}] {{(pid=81437) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:52:30.681798 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/tokens' {}] {{(pid=81438) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:52:30.681995 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/tokens' {}] {{(pid=81437) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:52:30.691346 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/saml2' {}] {{(pid=81438) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:52:30.691589 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/saml2' {}] {{(pid=81437) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:52:30.691677 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/saml2/ecp' {}] {{(pid=81438) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:52:30.691873 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/saml2/ecp' {}] {{(pid=81437) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:52:30.691956 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/websso/' {}] {{(pid=81438) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:52:30.692148 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/websso/' {}] {{(pid=81437) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:52:30.692241 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/identity_providers//protocols//websso' {}] {{(pid=81438) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:52:30.692433 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/identity_providers//protocols//websso' {}] {{(pid=81437) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:52:30.699401 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.common [-] Adding standard routes to API credentials for `type` (API Prefix: ) [/credentials, /credentials/] {{(pid=81438) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 13 15:52:30.699529 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.common [-] Adding standard routes to API credentials for `type` (API Prefix: ) [/credentials, /credentials/] {{(pid=81437) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 13 15:52:30.702284 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.common [-] Adding standard routes to API domains for `type` (API Prefix: ) [/domains, /domains/] {{(pid=81438) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 13 15:52:30.702382 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.common [-] Adding standard routes to API domains for `type` (API Prefix: ) [/domains, /domains/] {{(pid=81437) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 13 15:52:30.702575 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config' {}] {{(pid=81438) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:52:30.702660 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config' {}] {{(pid=81437) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:52:30.702865 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config/' {}] {{(pid=81438) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:52:30.702986 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config/' {}] {{(pid=81437) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:52:30.703177 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config//' {}] {{(pid=81438) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:52:30.703274 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config//' {}] {{(pid=81437) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:52:30.703467 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config/default' {}] {{(pid=81438) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:52:30.703550 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config/default' {}] {{(pid=81437) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:52:30.704006 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config//default' {}] {{(pid=81438) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:52:30.704090 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config//default' {}] {{(pid=81437) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:52:30.704164 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config///default' {}] {{(pid=81438) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:52:30.704228 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config///default' {}] {{(pid=81437) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:52:30.704289 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//users//roles' {}] {{(pid=81438) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:52:30.704349 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//users//roles' {}] {{(pid=81437) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:52:30.704529 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//users//roles/' {}] {{(pid=81438) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:52:30.704623 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//users//roles/' {}] {{(pid=81437) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:52:30.704805 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//groups//roles' {}] {{(pid=81438) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:52:30.704888 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//groups//roles' {}] {{(pid=81437) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:52:30.705074 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//groups//roles/' {}] {{(pid=81437) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:52:30.705155 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//groups//roles/' {}] {{(pid=81438) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:52:30.728059 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.common [-] Adding resource routes to API ec2tokens: ['/ec2tokens' {}] {{(pid=81437) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:52:30.728927 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.common [-] Adding resource routes to API ec2tokens: ['/ec2tokens' {}] {{(pid=81438) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:52:30.730061 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.common [-] Adding standard routes to API endpoints for `type` (API Prefix: ) [/endpoints, /endpoints/] {{(pid=81437) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 13 15:52:30.730342 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.common [-] Adding resource routes to API endpoints: ['/endpoints//OS-ENDPOINT-POLICY/policy' {}] {{(pid=81437) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:52:30.730428 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.common [-] Adding standard routes to API endpoints for `type` (API Prefix: ) [/endpoints, /endpoints/] {{(pid=81438) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 13 15:52:30.730659 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.common [-] Adding resource routes to API endpoints: ['/endpoints//OS-ENDPOINT-POLICY/policy' {}] {{(pid=81438) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:52:30.735006 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.common [-] Adding standard routes to API groups for `type` (API Prefix: ) [/groups, /groups/] {{(pid=81437) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 13 15:52:30.735299 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.common [-] Adding resource routes to API groups: ['/groups//users' {}] {{(pid=81437) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:52:30.735382 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.common [-] Adding standard routes to API groups for `type` (API Prefix: ) [/groups, /groups/] {{(pid=81438) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 13 15:52:30.735586 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.common [-] Adding resource routes to API groups: ['/groups//users/' {}] {{(pid=81437) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:52:30.735671 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.common [-] Adding resource routes to API groups: ['/groups//users' {}] {{(pid=81438) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:52:30.735880 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.common [-] Adding resource routes to API groups: ['/groups//users/' {}] {{(pid=81438) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:52:30.741912 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.common [-] Adding standard routes to API limits for `type` (API Prefix: ) [/limits, /limits/] {{(pid=81437) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 13 15:52:30.742161 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.common [-] Adding standard routes to API limits for `type` (API Prefix: ) [/limits, /limits/] {{(pid=81438) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 13 15:52:30.742246 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.common [-] Adding resource routes to API limits: ['/limits/model' {}] {{(pid=81437) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:52:30.742455 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.common [-] Adding resource routes to API limits: ['/limits/model' {}] {{(pid=81438) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:52:30.748069 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.common [-] Adding standard routes to API OS-EP-FILTER for `type` (API Prefix: /OS-EP-FILTER) [/endpoint_groups, /endpoint_groups/] {{(pid=81437) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 13 15:52:30.748182 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.common [-] Adding standard routes to API OS-EP-FILTER for `type` (API Prefix: /OS-EP-FILTER) [/endpoint_groups, /endpoint_groups/] {{(pid=81438) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 13 15:52:30.748374 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoints//projects' {}] {{(pid=81437) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:52:30.748458 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoints//projects' {}] {{(pid=81438) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:52:30.748669 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoints/' {}] {{(pid=81437) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:52:30.748752 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoints/' {}] {{(pid=81438) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:52:30.748953 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoints' {}] {{(pid=81438) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:52:30.749036 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoints' {}] {{(pid=81437) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:52:30.749229 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoint_groups' {}] {{(pid=81437) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:52:30.749312 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoint_groups' {}] {{(pid=81438) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:52:30.749505 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//endpoints' {}] {{(pid=81438) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:52:30.749596 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//endpoints' {}] {{(pid=81437) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:52:30.749775 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//projects' {}] {{(pid=81437) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:52:30.749856 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//projects' {}] {{(pid=81438) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:52:30.750048 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//projects/' {}] {{(pid=81437) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:52:30.750129 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//projects/' {}] {{(pid=81438) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:52:30.765538 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-FEDERATION: ['/saml2/metadata' {}] {{(pid=81438) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:52:30.765648 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-FEDERATION: ['/saml2/metadata' {}] {{(pid=81437) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:52:30.765826 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-FEDERATION: ['/identity_providers//protocols//auth' {}] {{(pid=81438) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:52:30.765912 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-FEDERATION: ['/identity_providers//protocols//auth' {}] {{(pid=81437) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:52:30.769678 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.common [-] Adding standard routes to API identity_providers for `type` (API Prefix: /OS-FEDERATION) [/identity_providers, /identity_providers/] {{(pid=81438) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 13 15:52:30.769771 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.common [-] Adding standard routes to API identity_providers for `type` (API Prefix: /OS-FEDERATION) [/identity_providers, /identity_providers/] {{(pid=81437) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 13 15:52:30.772928 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.common [-] Adding resource routes to API protocols: ['/OS-FEDERATION/identity_providers//protocols/' {}] {{(pid=81437) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:52:30.773014 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.common [-] Adding resource routes to API protocols: ['/OS-FEDERATION/identity_providers//protocols/' {}] {{(pid=81438) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:52:30.773224 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.common [-] Adding resource routes to API protocols: ['/OS-FEDERATION/identity_providers//protocols' {}] {{(pid=81438) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:52:30.773307 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.common [-] Adding resource routes to API protocols: ['/OS-FEDERATION/identity_providers//protocols' {}] {{(pid=81437) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:52:30.777990 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.common [-] Adding standard routes to API mappings for `type` (API Prefix: /OS-FEDERATION) [/mappings, /mappings/] {{(pid=81438) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 13 15:52:30.778105 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.common [-] Adding standard routes to API mappings for `type` (API Prefix: /OS-FEDERATION) [/mappings, /mappings/] {{(pid=81437) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 13 15:52:30.781134 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.common [-] Adding standard routes to API service_providers for `type` (API Prefix: /OS-FEDERATION) [/service_providers, /service_providers/] {{(pid=81437) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 13 15:52:30.781222 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.common [-] Adding standard routes to API service_providers for `type` (API Prefix: /OS-FEDERATION) [/service_providers, /service_providers/] {{(pid=81438) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 13 15:52:30.784352 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//groups//roles//inherited_to_projects' {}] {{(pid=81438) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:52:30.784441 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//groups//roles//inherited_to_projects' {}] {{(pid=81437) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:52:30.784658 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//groups//roles/inherited_to_projects' {}] {{(pid=81438) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:52:30.784744 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//groups//roles/inherited_to_projects' {}] {{(pid=81437) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:52:30.784948 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//users//roles//inherited_to_projects' {}] {{(pid=81438) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:52:30.785033 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//users//roles//inherited_to_projects' {}] {{(pid=81437) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:52:30.785244 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//users//roles/inherited_to_projects' {}] {{(pid=81438) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:52:30.785337 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//users//roles/inherited_to_projects' {}] {{(pid=81437) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:52:30.785536 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/projects//users//roles//inherited_to_projects' {}] {{(pid=81437) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:52:30.785619 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/projects//users//roles//inherited_to_projects' {}] {{(pid=81438) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:52:30.785825 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/projects//groups//roles//inherited_to_projects' {}] {{(pid=81438) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:52:30.785908 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/projects//groups//roles//inherited_to_projects' {}] {{(pid=81437) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:52:30.801593 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.common [-] Adding standard routes to API OS-OAUTH1 for `type` (API Prefix: /OS-OAUTH1) [/consumers, /consumers/] {{(pid=81438) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 13 15:52:30.801742 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.common [-] Adding standard routes to API OS-OAUTH1 for `type` (API Prefix: /OS-OAUTH1) [/consumers, /consumers/] {{(pid=81437) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 13 15:52:30.801883 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/request_token' {}] {{(pid=81438) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:52:30.802034 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/request_token' {}] {{(pid=81437) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:52:30.802154 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/access_token' {}] {{(pid=81438) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:52:30.802308 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/access_token' {}] {{(pid=81437) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:52:30.802415 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/authorize/' {}] {{(pid=81438) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:52:30.802579 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/authorize/' {}] {{(pid=81437) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:52:30.810126 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH2: ['/token' {}] {{(pid=81438) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:52:30.810239 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH2: ['/token' {}] {{(pid=81437) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:52:30.811709 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.common [-] Adding resource routes to API events: ['/events' {}] {{(pid=81438) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:52:30.811847 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.common [-] Adding resource routes to API events: ['/events' {}] {{(pid=81437) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:52:30.813209 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-SIMPLE-CERT: ['/OS-SIMPLE-CERT/ca' {}] {{(pid=81438) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:52:30.813304 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-SIMPLE-CERT: ['/OS-SIMPLE-CERT/ca' {}] {{(pid=81437) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:52:30.813496 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-SIMPLE-CERT: ['/OS-SIMPLE-CERT/certificates' {}] {{(pid=81438) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:52:30.813580 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-SIMPLE-CERT: ['/OS-SIMPLE-CERT/certificates' {}] {{(pid=81437) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:52:30.816218 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.common [-] Adding standard routes to API policy for `type` (API Prefix: ) [/policies, /policies/] {{(pid=81438) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 13 15:52:30.816318 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.common [-] Adding standard routes to API policy for `type` (API Prefix: ) [/policies, /policies/] {{(pid=81437) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 13 15:52:30.816509 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/endpoints' {}] {{(pid=81438) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:52:30.816592 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/endpoints' {}] {{(pid=81437) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:52:30.816800 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/endpoints/' {}] {{(pid=81438) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:52:30.816883 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/endpoints/' {}] {{(pid=81437) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:52:30.817087 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/services/' {}] {{(pid=81438) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:52:30.817171 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/services/' {}] {{(pid=81437) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:52:30.817478 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/services//regions/' {}] {{(pid=81438) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:52:30.817576 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/services//regions/' {}] {{(pid=81437) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:52:30.829137 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.common [-] Adding standard routes to API projects for `type` (API Prefix: ) [/projects, /projects/] {{(pid=81438) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 13 15:52:30.829258 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.common [-] Adding standard routes to API projects for `type` (API Prefix: ) [/projects, /projects/] {{(pid=81437) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 13 15:52:30.829424 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//tags' {}] {{(pid=81438) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:52:30.829507 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//tags' {}] {{(pid=81437) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:52:30.829710 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//tags/' {}] {{(pid=81437) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:52:30.829801 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//tags/' {}] {{(pid=81438) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:52:30.829994 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//users//roles/' {}] {{(pid=81438) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:52:30.830076 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//users//roles/' {}] {{(pid=81437) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:52:30.830377 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//users//roles' {}] {{(pid=81437) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:52:30.830458 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//users//roles' {}] {{(pid=81438) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:52:30.830666 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//groups//roles/' {}] {{(pid=81437) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:52:30.830756 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//groups//roles/' {}] {{(pid=81438) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:52:30.830972 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//groups//roles' {}] {{(pid=81438) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:52:30.831056 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//groups//roles' {}] {{(pid=81437) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:52:30.846342 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.common [-] Adding standard routes to API regions for `type` (API Prefix: ) [/regions, /regions/] {{(pid=81438) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 13 15:52:30.846441 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.common [-] Adding standard routes to API regions for `type` (API Prefix: ) [/regions, /regions/] {{(pid=81437) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 13 15:52:30.849269 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.common [-] Adding standard routes to API registered_limit for `type` (API Prefix: ) [/registered_limits, /registered_limits/] {{(pid=81438) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 13 15:52:30.849365 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.common [-] Adding standard routes to API registered_limit for `type` (API Prefix: ) [/registered_limits, /registered_limits/] {{(pid=81437) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 13 15:52:30.852274 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.common [-] Adding resource routes to API role_assignments: ['/role_assignments' {}] {{(pid=81437) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:52:30.852358 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.common [-] Adding resource routes to API role_assignments: ['/role_assignments' {}] {{(pid=81438) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:52:30.853902 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.common [-] Adding resource routes to API role_inferences: ['/role_inferences' {}] {{(pid=81437) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:52:30.854008 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.common [-] Adding resource routes to API role_inferences: ['/role_inferences' {}] {{(pid=81438) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:52:30.855396 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.common [-] Adding standard routes to API roles for `type` (API Prefix: ) [/roles, /roles/] {{(pid=81437) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 13 15:52:30.855487 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.common [-] Adding standard routes to API roles for `type` (API Prefix: ) [/roles, /roles/] {{(pid=81438) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 13 15:52:30.855679 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.common [-] Adding resource routes to API roles: ['/roles//implies' {}] {{(pid=81438) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:52:30.855760 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.common [-] Adding resource routes to API roles: ['/roles//implies' {}] {{(pid=81437) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:52:30.856078 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.common [-] Adding resource routes to API roles: ['/roles//implies/' {}] {{(pid=81438) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:52:30.856209 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.common [-] Adding resource routes to API roles: ['/roles//implies/' {}] {{(pid=81437) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:52:30.862420 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.common [-] Adding resource routes to API s3tokens: ['/s3tokens' {}] {{(pid=81438) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:52:30.862514 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.common [-] Adding resource routes to API s3tokens: ['/s3tokens' {}] {{(pid=81437) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:52:30.863860 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.common [-] Adding standard routes to API services for `type` (API Prefix: ) [/services, /services/] {{(pid=81438) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 13 15:52:30.863945 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.common [-] Adding standard routes to API services for `type` (API Prefix: ) [/services, /services/] {{(pid=81437) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 13 15:52:30.866722 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/users//roles' {}] {{(pid=81438) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:52:30.866806 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/users//roles' {}] {{(pid=81437) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:52:30.867143 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/users//roles/' {}] {{(pid=81438) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:52:30.867280 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/users//roles/' {}] {{(pid=81437) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:52:30.867439 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/groups//roles' {}] {{(pid=81438) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:52:30.867577 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/groups//roles' {}] {{(pid=81437) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:52:30.867724 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/groups//roles/' {}] {{(pid=81438) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:52:30.867851 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/groups//roles/' {}] {{(pid=81437) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:52:30.876063 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.common [-] Adding standard routes to API trusts for `type` (API Prefix: /OS-TRUST) [/trusts, /trusts/] {{(pid=81438) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 13 15:52:30.876171 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.common [-] Adding standard routes to API trusts for `type` (API Prefix: /OS-TRUST) [/trusts, /trusts/] {{(pid=81437) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 13 15:52:30.876355 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.common [-] Adding resource routes to API trusts: ['/trusts//roles' {}] {{(pid=81438) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:52:30.876446 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.common [-] Adding resource routes to API trusts: ['/trusts//roles' {}] {{(pid=81437) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:52:30.876641 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.common [-] Adding resource routes to API trusts: ['/trusts//roles/' {}] {{(pid=81438) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:52:30.876736 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.common [-] Adding resource routes to API trusts: ['/trusts//roles/' {}] {{(pid=81437) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:52:30.883499 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.common [-] Adding standard routes to API users for `type` (API Prefix: ) [/users, /users/] {{(pid=81438) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 13 15:52:30.883594 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.common [-] Adding standard routes to API users for `type` (API Prefix: ) [/users, /users/] {{(pid=81437) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 13 15:52:30.883784 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//password' {}] {{(pid=81438) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:52:30.883867 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//password' {}] {{(pid=81437) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:52:30.884067 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//groups' {}] {{(pid=81438) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:52:30.884150 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//groups' {}] {{(pid=81437) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:52:30.884349 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//projects' {}] {{(pid=81438) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:52:30.884504 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//projects' {}] {{(pid=81437) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:52:30.885255 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//credentials/OS-EC2' {}] {{(pid=81438) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:52:30.885339 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//credentials/OS-EC2' {}] {{(pid=81437) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:52:30.885536 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//credentials/OS-EC2/' {}] {{(pid=81437) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:52:30.885617 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//credentials/OS-EC2/' {}] {{(pid=81438) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:52:30.885818 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens' {}] {{(pid=81437) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:52:30.885907 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens' {}] {{(pid=81438) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:52:30.886107 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens/' {}] {{(pid=81437) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:52:30.886197 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens/' {}] {{(pid=81438) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:52:30.886382 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens//roles' {}] {{(pid=81437) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:52:30.886471 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens//roles' {}] {{(pid=81438) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:52:30.886675 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens//roles/' {}] {{(pid=81437) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:52:30.886757 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens//roles/' {}] {{(pid=81438) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:52:30.886989 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//application_credentials' {}] {{(pid=81437) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:52:30.887074 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//application_credentials' {}] {{(pid=81438) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:52:30.887260 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//application_credentials/' {}] {{(pid=81437) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:52:30.887344 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//application_credentials/' {}] {{(pid=81438) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:52:30.887535 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//access_rules' {}] {{(pid=81438) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:52:30.887614 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//access_rules' {}] {{(pid=81437) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:52:30.887804 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//access_rules/' {}] {{(pid=81437) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:52:30.887884 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//access_rules/' {}] {{(pid=81438) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:52:30.921611 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: WSGI app 0 (mountpoint='') ready in 5 seconds on interpreter 0x5623371229c0 pid: 81437 (default app) Sep 13 15:52:30.921719 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: WSGI app 0 (mountpoint='') ready in 5 seconds on interpreter 0x5623371229c0 pid: 81438 (default app) Sep 13 15:52:30.924930 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd90e41d-443c-43a0-8edd-bcf4c01943de None None] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:52:30.925219 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd90e41d-443c-43a0-8edd-bcf4c01943de None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:52:30.925495 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd90e41d-443c-43a0-8edd-bcf4c01943de None None] PATH_INFO: `/v3` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:52:30.929940 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1/1] 10.222.0.50 () {50 vars in 791 bytes} [Tue Sep 13 15:52:30 2022] GET /identity/v3/ => generated 255 bytes in 8 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:52:37.742692 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8f6a46e-b8a6-4cd4-9ea5-215546abd517 None None] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:52:37.743235 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8f6a46e-b8a6-4cd4-9ea5-215546abd517 None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:52:37.743341 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8f6a46e-b8a6-4cd4-9ea5-215546abd517 None None] PATH_INFO: `/` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:52:37.747914 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1/2] 10.222.0.50 () {54 vars in 927 bytes} [Tue Sep 13 15:52:37 2022] GET /identity => generated 270 bytes in 9 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 13 15:52:37.753267 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73a29f16-e217-4b43-8f59-a75487fc7dbb None None] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:52:37.753816 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73a29f16-e217-4b43-8f59-a75487fc7dbb None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:52:37.753816 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73a29f16-e217-4b43-8f59-a75487fc7dbb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:52:37.824513 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: INFO dbcounter [None req-73a29f16-e217-4b43-8f59-a75487fc7dbb None None] Registered counter for database keystone Sep 13 15:52:37.843785 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG oslo_db.sqlalchemy.engines [None req-73a29f16-e217-4b43-8f59-a75487fc7dbb None None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=81437) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Sep 13 15:52:37.849889 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writer thread running {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:99}} Sep 13 15:52:38.028910 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG passlib.handlers.bcrypt [None req-73a29f16-e217-4b43-8f59-a75487fc7dbb None None] detected 'bcrypt' backend, version '4.0.0' {{(pid=81437) _load_backend_mixin /usr/local/lib/python3.8/dist-packages/passlib/handlers/bcrypt.py:625}} Sep 13 15:52:38.029240 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG passlib.handlers.bcrypt [None req-73a29f16-e217-4b43-8f59-a75487fc7dbb None None] 'bcrypt' backend lacks $2$ support, enabling workaround {{(pid=81437) _finalize_backend_mixin /usr/local/lib/python3.8/dist-packages/passlib/handlers/bcrypt.py:406}} Sep 13 15:52:38.087907 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.auth.core [None req-73a29f16-e217-4b43-8f59-a75487fc7dbb None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81437) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:52:38.089535 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-73a29f16-e217-4b43-8f59-a75487fc7dbb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:52:38.215936 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 2/3] 10.222.0.50 () {58 vars in 1024 bytes} [Tue Sep 13 15:52:37 2022] POST /identity/v3/auth/tokens => generated 914 bytes in 464 msecs (HTTP/1.1 201) 6 headers in 384 bytes (2 switches on core 0) Sep 13 15:52:38.716410 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8f6a46e-b8a6-4cd4-9ea5-215546abd517 None None] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:52:38.786708 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: INFO dbcounter [None req-c8f6a46e-b8a6-4cd4-9ea5-215546abd517 None None] Registered counter for database keystone Sep 13 15:52:38.809430 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG oslo_db.sqlalchemy.engines [None req-c8f6a46e-b8a6-4cd4-9ea5-215546abd517 None None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=81438) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Sep 13 15:52:38.815641 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writer thread running {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:99}} Sep 13 15:52:38.937880 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8f6a46e-b8a6-4cd4-9ea5-215546abd517 None None] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:52:38.942097 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c495207-404b-4638-86d3-b20018eeb2a1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:52:38.942715 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c495207-404b-4638-86d3-b20018eeb2a1 admin admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:52:38.943021 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c495207-404b-4638-86d3-b20018eeb2a1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:52:38.943292 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c495207-404b-4638-86d3-b20018eeb2a1 admin admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:52:38.952317 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c495207-404b-4638-86d3-b20018eeb2a1 admin admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:52:39.003283 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: WARNING keystone.common.rbac_enforcer.enforcer [None req-4c495207-404b-4638-86d3-b20018eeb2a1 admin admin] Deprecated policy rules found. Use oslopolicy-policy-generator and oslopolicy-policy-upgrade to detect and resolve deprecated policies in your configuration. Sep 13 15:52:39.003566 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c495207-404b-4638-86d3-b20018eeb2a1 admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:52:39.013410 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 2/4] 10.222.0.50 () {58 vars in 1324 bytes} [Tue Sep 13 15:52:38 2022] GET /identity/v3/auth/tokens => generated 914 bytes in 298 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Sep 13 15:52:39.019354 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73a29f16-e217-4b43-8f59-a75487fc7dbb None None] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:52:39.036025 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73a29f16-e217-4b43-8f59-a75487fc7dbb None None] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:52:39.040488 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb4d3605-5ce3-408c-b17a-09953ecf3726 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:52:39.041643 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb4d3605-5ce3-408c-b17a-09953ecf3726 admin admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:52:39.041926 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb4d3605-5ce3-408c-b17a-09953ecf3726 admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:52:39.042198 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb4d3605-5ce3-408c-b17a-09953ecf3726 admin admin] PATH_INFO: `/v3/projects/555bee08a3fa4d08bdd8ecc881adc45b` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:52:39.043838 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb4d3605-5ce3-408c-b17a-09953ecf3726 admin admin] RBAC: Authorizing `identity:get_project(project_id=555bee08a3fa4d08bdd8ecc881adc45b)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:52:39.096994 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: WARNING keystone.common.rbac_enforcer.enforcer [None req-eb4d3605-5ce3-408c-b17a-09953ecf3726 admin admin] Deprecated policy rules found. Use oslopolicy-policy-generator and oslopolicy-policy-upgrade to detect and resolve deprecated policies in your configuration. Sep 13 15:52:39.097342 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb4d3605-5ce3-408c-b17a-09953ecf3726 admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:52:39.101942 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 3/5] 10.222.0.50 () {56 vars in 1207 bytes} [Tue Sep 13 15:52:39 2022] GET /identity/v3/projects/555bee08a3fa4d08bdd8ecc881adc45b => generated 344 bytes in 83 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:52:41.227114 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d8aebab-c6aa-41f4-a75d-b53d906ff78e None None] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:52:41.227672 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d8aebab-c6aa-41f4-a75d-b53d906ff78e None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:52:41.227672 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d8aebab-c6aa-41f4-a75d-b53d906ff78e None None] PATH_INFO: `/` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:52:41.232020 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 3/6] 10.222.0.50 () {54 vars in 927 bytes} [Tue Sep 13 15:52:41 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:52:41.237547 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a0543b9-277e-45ad-a300-224ac0b3dbe7 None None] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:52:41.237940 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a0543b9-277e-45ad-a300-224ac0b3dbe7 None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:52:41.238098 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a0543b9-277e-45ad-a300-224ac0b3dbe7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:52:41.257691 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.auth.core [None req-7a0543b9-277e-45ad-a300-224ac0b3dbe7 None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81437) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:52:41.259226 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-7a0543b9-277e-45ad-a300-224ac0b3dbe7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:52:41.267536 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 4/7] 10.222.0.50 () {58 vars in 1024 bytes} [Tue Sep 13 15:52:41 2022] POST /identity/v3/auth/tokens => generated 914 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 384 bytes (2 switches on core 0) Sep 13 15:52:41.538345 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d8aebab-c6aa-41f4-a75d-b53d906ff78e None None] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:52:41.550482 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d8aebab-c6aa-41f4-a75d-b53d906ff78e None None] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:52:41.554690 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0b0d923-baa1-4b99-9e7e-6ecf85e590f0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:52:41.555582 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0b0d923-baa1-4b99-9e7e-6ecf85e590f0 admin admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:52:41.555862 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0b0d923-baa1-4b99-9e7e-6ecf85e590f0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:52:41.556138 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0b0d923-baa1-4b99-9e7e-6ecf85e590f0 admin admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:52:41.564275 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0b0d923-baa1-4b99-9e7e-6ecf85e590f0 admin admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:52:41.566051 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0b0d923-baa1-4b99-9e7e-6ecf85e590f0 admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:52:41.575211 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 4/8] 10.222.0.50 () {58 vars in 1324 bytes} [Tue Sep 13 15:52:41 2022] GET /identity/v3/auth/tokens => generated 914 bytes in 37 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Sep 13 15:52:41.580528 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a0543b9-277e-45ad-a300-224ac0b3dbe7 None None] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:52:41.589595 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a0543b9-277e-45ad-a300-224ac0b3dbe7 None None] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:52:41.593939 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-428ac724-9470-4d45-8ff6-33b7a9665982 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:52:41.594559 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-428ac724-9470-4d45-8ff6-33b7a9665982 admin admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:52:41.594841 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-428ac724-9470-4d45-8ff6-33b7a9665982 admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:52:41.595152 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-428ac724-9470-4d45-8ff6-33b7a9665982 admin admin] PATH_INFO: `/v3/users/e0a915a0eb3c42c7af97dcc7eed5b42f` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:52:41.596445 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-428ac724-9470-4d45-8ff6-33b7a9665982 admin admin] RBAC: Authorizing `identity:get_user(user_id=e0a915a0eb3c42c7af97dcc7eed5b42f)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:52:41.598267 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-428ac724-9470-4d45-8ff6-33b7a9665982 admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:52:41.599819 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 5/9] 10.222.0.50 () {56 vars in 1198 bytes} [Tue Sep 13 15:52:41 2022] GET /identity/v3/users/e0a915a0eb3c42c7af97dcc7eed5b42f => generated 247 bytes in 20 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:52:43.817694 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-516320dd-ec1c-4167-8650-913907727fa4 None None] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:52:43.818243 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-516320dd-ec1c-4167-8650-913907727fa4 None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:52:43.818338 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-516320dd-ec1c-4167-8650-913907727fa4 None None] PATH_INFO: `/` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:52:43.819774 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 5/10] 10.222.0.50 () {54 vars in 927 bytes} [Tue Sep 13 15:52:43 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:52:43.825399 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-343ff207-4ef7-4ba9-adec-e87487901e5a None None] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:52:43.825832 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97e07c67-5bca-481b-aa6e-5d77c9f150dd None None] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:52:43.825927 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-343ff207-4ef7-4ba9-adec-e87487901e5a None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:52:43.826008 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97e07c67-5bca-481b-aa6e-5d77c9f150dd None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:52:43.826170 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-343ff207-4ef7-4ba9-adec-e87487901e5a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:52:43.826257 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97e07c67-5bca-481b-aa6e-5d77c9f150dd None None] PATH_INFO: `/` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:52:43.827392 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 6/11] 10.222.0.50 () {54 vars in 927 bytes} [Tue Sep 13 15:52:43 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 13 15:52:43.832387 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-164980af-b8dc-42eb-8919-3e935560c231 None None] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:52:43.832675 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-164980af-b8dc-42eb-8919-3e935560c231 None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:52:43.832953 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-164980af-b8dc-42eb-8919-3e935560c231 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:52:43.845278 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.auth.core [None req-343ff207-4ef7-4ba9-adec-e87487901e5a None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81437) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:52:43.846460 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-343ff207-4ef7-4ba9-adec-e87487901e5a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:52:43.875343 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG passlib.handlers.bcrypt [None req-164980af-b8dc-42eb-8919-3e935560c231 None None] detected 'bcrypt' backend, version '4.0.0' {{(pid=81438) _load_backend_mixin /usr/local/lib/python3.8/dist-packages/passlib/handlers/bcrypt.py:625}} Sep 13 15:52:43.875751 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG passlib.handlers.bcrypt [None req-164980af-b8dc-42eb-8919-3e935560c231 None None] 'bcrypt' backend lacks $2$ support, enabling workaround {{(pid=81438) _finalize_backend_mixin /usr/local/lib/python3.8/dist-packages/passlib/handlers/bcrypt.py:406}} Sep 13 15:52:43.925022 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.auth.core [None req-164980af-b8dc-42eb-8919-3e935560c231 None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81438) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:52:43.926606 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-164980af-b8dc-42eb-8919-3e935560c231 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:52:43.927657 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 6/12] 10.222.0.50 () {58 vars in 1024 bytes} [Tue Sep 13 15:52:43 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 13 15:52:44.025218 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 7/13] 10.222.0.50 () {58 vars in 1024 bytes} [Tue Sep 13 15:52:43 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 193 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 13 15:52:44.196506 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-343ff207-4ef7-4ba9-adec-e87487901e5a None None] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:52:44.222085 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:52:44.293817 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-164980af-b8dc-42eb-8919-3e935560c231 None None] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:52:44.303313 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-343ff207-4ef7-4ba9-adec-e87487901e5a None None] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:52:44.337414 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e87b82d-faeb-4af0-a231-85634a900f05 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:52:44.338063 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e87b82d-faeb-4af0-a231-85634a900f05 None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:52:44.338409 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e87b82d-faeb-4af0-a231-85634a900f05 None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:52:44.338686 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e87b82d-faeb-4af0-a231-85634a900f05 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:52:44.349908 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e87b82d-faeb-4af0-a231-85634a900f05 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 13 15:52:44.359594 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application [None req-4e87b82d-faeb-4af0-a231-85634a900f05 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 13 15:52:44.359594 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:52:44.359594 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:52:44.359594 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:52:44.359594 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:52:44.359594 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:52:44.359594 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:52:44.359594 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:52:44.359594 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:52:44.359594 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:52:44.359594 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:52:44.359594 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:52:44.359594 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 13 15:52:44.359594 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 13 15:52:44.359594 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 13 15:52:44.359594 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 13 15:52:44.360615 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 13 15:52:44.360615 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 13 15:52:44.360615 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 13 15:52:44.360615 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 13 15:52:44.360615 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:52:44.360615 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:52:44.360615 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:52:44.360615 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:52:44.360615 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:52:44.360615 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:52:44.360615 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:52:44.360615 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application with Lock( Sep 13 15:52:44.360615 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:52:44.360615 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:52:44.360615 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:52:44.360615 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:52:44.361824 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:52:44.361824 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:52:44.361824 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:52:44.361824 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:52:44.361824 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 13 15:52:44.361824 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 13 15:52:44.361824 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 13 15:52:44.361824 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 13 15:52:44.361824 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 13 15:52:44.361824 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 13 15:52:44.361824 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 13 15:52:44.361824 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application Sep 13 15:52:44.361824 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 7/14] 10.222.0.50 () {56 vars in 1096 bytes} [Tue Sep 13 15:52:44 2022] GET /identity/v3/roles/admin => generated 83 bytes in 165 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:52:44.366512 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e87b82d-faeb-4af0-a231-85634a900f05 None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:52:44.397081 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-164980af-b8dc-42eb-8919-3e935560c231 None None] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:52:44.399356 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:52:44.434651 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-554d9f14-7d6d-4b97-9ca8-70aed44b23e8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:52:44.435547 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-554d9f14-7d6d-4b97-9ca8-70aed44b23e8 None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:52:44.435831 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-554d9f14-7d6d-4b97-9ca8-70aed44b23e8 None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:52:44.436107 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-554d9f14-7d6d-4b97-9ca8-70aed44b23e8 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:52:44.449622 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:52:44.468464 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e87b82d-faeb-4af0-a231-85634a900f05 None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:52:44.502200 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d7942e4-21d7-41aa-9007-27a1801a9cc3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:52:44.502805 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d7942e4-21d7-41aa-9007-27a1801a9cc3 None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:52:44.503110 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d7942e4-21d7-41aa-9007-27a1801a9cc3 None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:52:44.503382 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d7942e4-21d7-41aa-9007-27a1801a9cc3 None admin] PATH_INFO: `/v3/roles` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:52:44.504039 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d7942e4-21d7-41aa-9007-27a1801a9cc3 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81437) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:52:44.504323 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-554d9f14-7d6d-4b97-9ca8-70aed44b23e8 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:52:44.504431 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d7942e4-21d7-41aa-9007-27a1801a9cc3 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:52:44.505727 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d7942e4-21d7-41aa-9007-27a1801a9cc3 None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:52:44.505819 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-554d9f14-7d6d-4b97-9ca8-70aed44b23e8 None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:52:44.517115 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 8/15] 10.222.0.50 () {56 vars in 1099 bytes} [Tue Sep 13 15:52:44 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 13 15:52:44.522762 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d7942e4-21d7-41aa-9007-27a1801a9cc3 None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:52:44.604856 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 8/16] 10.222.0.50 () {58 vars in 1282 bytes} [Tue Sep 13 15:52:44 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 312 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Sep 13 15:52:44.610443 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-554d9f14-7d6d-4b97-9ca8-70aed44b23e8 None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:52:44.613160 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:52:44.623275 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d7942e4-21d7-41aa-9007-27a1801a9cc3 None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:52:44.656188 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82bb8235-6cee-432d-82e0-496977d67bfc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:52:44.656813 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82bb8235-6cee-432d-82e0-496977d67bfc None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:52:44.657093 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82bb8235-6cee-432d-82e0-496977d67bfc None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:52:44.657366 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82bb8235-6cee-432d-82e0-496977d67bfc None admin] PATH_INFO: `/v3/users/e0a915a0eb3c42c7af97dcc7eed5b42f` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:52:44.658492 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82bb8235-6cee-432d-82e0-496977d67bfc None admin] RBAC: Authorizing `identity:get_user(user_id=e0a915a0eb3c42c7af97dcc7eed5b42f)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:52:44.660284 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82bb8235-6cee-432d-82e0-496977d67bfc None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:52:44.661836 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 9/17] 10.222.0.50 () {56 vars in 1177 bytes} [Tue Sep 13 15:52:44 2022] GET /identity/v3/users/e0a915a0eb3c42c7af97dcc7eed5b42f => generated 247 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 13 15:52:44.667315 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82bb8235-6cee-432d-82e0-496977d67bfc None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:52:44.681778 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:52:44.710403 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-554d9f14-7d6d-4b97-9ca8-70aed44b23e8 None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:52:44.746046 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d9b3eff-d196-49d6-ba4d-86db16996e21 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:52:44.746934 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d9b3eff-d196-49d6-ba4d-86db16996e21 None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:52:44.747226 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d9b3eff-d196-49d6-ba4d-86db16996e21 None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:52:44.747506 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d9b3eff-d196-49d6-ba4d-86db16996e21 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:52:44.766347 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82bb8235-6cee-432d-82e0-496977d67bfc None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:52:44.766477 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d9b3eff-d196-49d6-ba4d-86db16996e21 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:52:44.767863 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d9b3eff-d196-49d6-ba4d-86db16996e21 None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:52:44.785727 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application [None req-0d9b3eff-d196-49d6-ba4d-86db16996e21 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:52:44.785727 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:52:44.785727 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:52:44.785727 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:52:44.785727 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:52:44.785727 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:52:44.785727 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:52:44.785727 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:52:44.785727 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:52:44.785727 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:52:44.785727 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:52:44.785727 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:52:44.785727 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 13 15:52:44.785727 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 13 15:52:44.785727 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 13 15:52:44.785727 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 13 15:52:44.786640 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:52:44.786640 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:52:44.786640 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:52:44.786640 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:52:44.786640 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:52:44.786640 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:52:44.786640 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:52:44.786640 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application with Lock( Sep 13 15:52:44.786640 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:52:44.786640 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:52:44.786640 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:52:44.786640 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:52:44.786640 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:52:44.786640 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:52:44.786640 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:52:44.786640 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:52:44.786640 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 13 15:52:44.787468 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 13 15:52:44.787468 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:52:44.787468 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Sep 13 15:52:44.787468 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 9/18] 10.222.0.50 () {56 vars in 1108 bytes} [Tue Sep 13 15:52:44 2022] GET /identity/v3/domains/Default => generated 87 bytes in 177 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 13 15:52:44.792687 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d9b3eff-d196-49d6-ba4d-86db16996e21 None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:52:44.802272 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff81d410-312f-41f4-b50d-bcb8e2d33b37 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:52:44.802944 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff81d410-312f-41f4-b50d-bcb8e2d33b37 None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:52:44.803227 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff81d410-312f-41f4-b50d-bcb8e2d33b37 None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:52:44.803498 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff81d410-312f-41f4-b50d-bcb8e2d33b37 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:52:44.804641 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff81d410-312f-41f4-b50d-bcb8e2d33b37 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:52:44.806374 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff81d410-312f-41f4-b50d-bcb8e2d33b37 None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:52:44.808489 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 10/19] 10.222.0.50 () {56 vars in 1108 bytes} [Tue Sep 13 15:52:44 2022] GET /identity/v3/domains/default => generated 199 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:52:44.814205 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff81d410-312f-41f4-b50d-bcb8e2d33b37 None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:52:44.849536 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:52:44.893026 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:52:44.894969 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d9b3eff-d196-49d6-ba4d-86db16996e21 None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:52:44.912790 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff81d410-312f-41f4-b50d-bcb8e2d33b37 None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:52:44.928699 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3933ba09-218a-4691-97ef-33fa45b4b000 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:52:44.929303 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3933ba09-218a-4691-97ef-33fa45b4b000 None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:52:44.929582 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3933ba09-218a-4691-97ef-33fa45b4b000 None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:52:44.929859 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3933ba09-218a-4691-97ef-33fa45b4b000 None admin] PATH_INFO: `/v3/domains` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:52:44.930745 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3933ba09-218a-4691-97ef-33fa45b4b000 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81438) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:52:44.931081 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3933ba09-218a-4691-97ef-33fa45b4b000 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:52:44.933189 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3933ba09-218a-4691-97ef-33fa45b4b000 None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:52:44.947982 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c97a6b3c-3cf0-4f2a-8163-6c9b1fe5c168 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:52:44.948599 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c97a6b3c-3cf0-4f2a-8163-6c9b1fe5c168 None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:52:44.948880 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c97a6b3c-3cf0-4f2a-8163-6c9b1fe5c168 None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:52:44.949158 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c97a6b3c-3cf0-4f2a-8163-6c9b1fe5c168 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:52:44.949887 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c97a6b3c-3cf0-4f2a-8163-6c9b1fe5c168 None admin] RBAC: Adding query filter params (role.id=031d4e1a23e34f71858fc4a6dfbd4ccb, scope.domain.id=default, user.id=e0a915a0eb3c42c7af97dcc7eed5b42f) {{(pid=81437) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:52:44.950199 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c97a6b3c-3cf0-4f2a-8163-6c9b1fe5c168 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:52:44.950802 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 10/20] 10.222.0.50 () {56 vars in 1109 bytes} [Tue Sep 13 15:52:44 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 13 15:52:44.951746 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c97a6b3c-3cf0-4f2a-8163-6c9b1fe5c168 None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:52:44.960523 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 11/21] 10.222.0.50 () {56 vars in 1322 bytes} [Tue Sep 13 15:52:44 2022] GET /identity/v3/role_assignments?user.id=e0a915a0eb3c42c7af97dcc7eed5b42f&scope.domain.id=default&role.id=031d4e1a23e34f71858fc4a6dfbd4ccb => generated 233 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 13 15:52:47.452921 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a5c006a-96eb-44d3-9dfc-c1d3e7979600 None None] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:52:47.453876 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a5c006a-96eb-44d3-9dfc-c1d3e7979600 None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:52:47.454356 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a5c006a-96eb-44d3-9dfc-c1d3e7979600 None None] PATH_INFO: `/` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:52:47.456153 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 11/22] 10.222.0.50 () {54 vars in 927 bytes} [Tue Sep 13 15:52:47 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:52:47.456588 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14fdac06-7c7d-4720-b951-31aafab5d760 None None] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:52:47.457478 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14fdac06-7c7d-4720-b951-31aafab5d760 None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:52:47.458049 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14fdac06-7c7d-4720-b951-31aafab5d760 None None] PATH_INFO: `/` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:52:47.459611 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 12/23] 10.222.0.50 () {54 vars in 927 bytes} [Tue Sep 13 15:52:47 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 13 15:52:47.461623 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2e3be9d-98ec-4c08-91c1-d247f5b0f2fa None None] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:52:47.461907 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2e3be9d-98ec-4c08-91c1-d247f5b0f2fa None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:52:47.462184 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2e3be9d-98ec-4c08-91c1-d247f5b0f2fa None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:52:47.476662 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a226372-180a-4a36-bace-95ef1067e776 None None] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:52:47.476940 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a226372-180a-4a36-bace-95ef1067e776 None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:52:47.477210 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a226372-180a-4a36-bace-95ef1067e776 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:52:47.511577 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.auth.core [None req-d2e3be9d-98ec-4c08-91c1-d247f5b0f2fa None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81438) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:52:47.511782 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.auth.core [None req-1a226372-180a-4a36-bace-95ef1067e776 None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81437) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:52:47.512882 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-1a226372-180a-4a36-bace-95ef1067e776 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:52:47.513067 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-d2e3be9d-98ec-4c08-91c1-d247f5b0f2fa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:52:47.672608 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 12/24] 10.222.0.50 () {58 vars in 1024 bytes} [Tue Sep 13 15:52:47 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 212 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 13 15:52:47.682922 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 13/25] 10.222.0.50 () {58 vars in 1024 bytes} [Tue Sep 13 15:52:47 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 207 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 13 15:52:47.930642 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08ff4d5e-ddb1-4b9e-9f1e-063229767bdd None None] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:52:47.931399 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08ff4d5e-ddb1-4b9e-9f1e-063229767bdd None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:52:47.931881 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08ff4d5e-ddb1-4b9e-9f1e-063229767bdd None None] PATH_INFO: `/` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:52:47.933825 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 13/26] 10.222.0.50 () {54 vars in 927 bytes} [Tue Sep 13 15:52:47 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:52:47.938929 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-364c6ef7-9297-4137-a8fc-32412026fd68 None None] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:52:47.939447 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-364c6ef7-9297-4137-a8fc-32412026fd68 None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:52:47.939940 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-364c6ef7-9297-4137-a8fc-32412026fd68 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:52:47.962668 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.auth.core [None req-364c6ef7-9297-4137-a8fc-32412026fd68 None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81437) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:52:47.964841 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-364c6ef7-9297-4137-a8fc-32412026fd68 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:52:48.053154 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 14/27] 10.222.0.50 () {58 vars in 1024 bytes} [Tue Sep 13 15:52:47 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 115 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 13 15:52:48.090171 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08ff4d5e-ddb1-4b9e-9f1e-063229767bdd None None] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:52:48.154251 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:52:48.160027 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-364c6ef7-9297-4137-a8fc-32412026fd68 None None] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:52:48.173287 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:52:48.271718 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08ff4d5e-ddb1-4b9e-9f1e-063229767bdd None None] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:52:48.340148 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-364c6ef7-9297-4137-a8fc-32412026fd68 None None] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:52:48.340335 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2098b9de-651b-44de-b666-e518c7b9783e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:52:48.340838 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2098b9de-651b-44de-b666-e518c7b9783e None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:52:48.341120 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2098b9de-651b-44de-b666-e518c7b9783e None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:52:48.341403 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2098b9de-651b-44de-b666-e518c7b9783e None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:52:48.373309 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2098b9de-651b-44de-b666-e518c7b9783e None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 13 15:52:48.379862 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application [None req-2098b9de-651b-44de-b666-e518c7b9783e None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 13 15:52:48.379862 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:52:48.379862 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:52:48.379862 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:52:48.379862 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:52:48.379862 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:52:48.379862 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:52:48.379862 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:52:48.379862 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:52:48.379862 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:52:48.379862 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:52:48.379862 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:52:48.379862 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 13 15:52:48.379862 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 13 15:52:48.379862 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 13 15:52:48.379862 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 13 15:52:48.380691 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 13 15:52:48.380691 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 13 15:52:48.380691 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 13 15:52:48.380691 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 13 15:52:48.380691 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:52:48.380691 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:52:48.380691 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:52:48.380691 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:52:48.380691 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:52:48.380691 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:52:48.380691 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:52:48.380691 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application with Lock( Sep 13 15:52:48.380691 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:52:48.380691 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:52:48.380691 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:52:48.380691 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:52:48.381457 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:52:48.381457 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:52:48.381457 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:52:48.381457 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:52:48.381457 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 13 15:52:48.381457 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 13 15:52:48.381457 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 13 15:52:48.381457 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 13 15:52:48.381457 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 13 15:52:48.381457 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 13 15:52:48.381457 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 13 15:52:48.381457 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Sep 13 15:52:48.381457 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 14/28] 10.222.0.50 () {56 vars in 1096 bytes} [Tue Sep 13 15:52:48 2022] GET /identity/v3/roles/admin => generated 83 bytes in 291 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:52:48.382840 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e07fd279-e7d2-4735-8c42-79d99075220d None None] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:52:48.382840 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e07fd279-e7d2-4735-8c42-79d99075220d None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:52:48.382840 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e07fd279-e7d2-4735-8c42-79d99075220d None None] PATH_INFO: `/` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:52:48.397474 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 15/29] 10.222.0.50 () {54 vars in 927 bytes} [Tue Sep 13 15:52:48 2022] GET /identity => generated 270 bytes in 16 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:52:48.404804 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e07fd279-e7d2-4735-8c42-79d99075220d None None] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:52:48.416317 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97bbce65-e6e7-4da9-85d2-a05f8f9816d1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:52:48.417750 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97bbce65-e6e7-4da9-85d2-a05f8f9816d1 None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:52:48.418035 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97bbce65-e6e7-4da9-85d2-a05f8f9816d1 None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:52:48.418312 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97bbce65-e6e7-4da9-85d2-a05f8f9816d1 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:52:48.430604 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97bbce65-e6e7-4da9-85d2-a05f8f9816d1 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:52:48.432135 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97bbce65-e6e7-4da9-85d2-a05f8f9816d1 None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:52:48.457431 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application [None req-97bbce65-e6e7-4da9-85d2-a05f8f9816d1 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:52:48.457431 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:52:48.457431 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:52:48.457431 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:52:48.457431 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:52:48.457431 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:52:48.457431 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:52:48.457431 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:52:48.457431 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:52:48.457431 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:52:48.457431 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:52:48.457431 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:52:48.457431 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 13 15:52:48.457431 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 13 15:52:48.457431 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 13 15:52:48.457431 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 13 15:52:48.459991 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:52:48.459991 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:52:48.459991 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:52:48.459991 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:52:48.459991 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:52:48.459991 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:52:48.459991 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:52:48.459991 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application with Lock( Sep 13 15:52:48.459991 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:52:48.459991 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:52:48.459991 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:52:48.459991 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:52:48.459991 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:52:48.459991 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:52:48.459991 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:52:48.459991 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:52:48.459991 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 13 15:52:48.462015 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 13 15:52:48.462015 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:52:48.462015 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application Sep 13 15:52:48.462015 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 15/30] 10.222.0.50 () {56 vars in 1108 bytes} [Tue Sep 13 15:52:48 2022] GET /identity/v3/domains/Default => generated 87 bytes in 299 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:52:48.462015 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f6ccda1-14cb-4351-9089-d5fa2ad16eee None None] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:52:48.462015 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f6ccda1-14cb-4351-9089-d5fa2ad16eee None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:52:48.462015 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f6ccda1-14cb-4351-9089-d5fa2ad16eee None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:52:48.504055 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.auth.core [None req-4f6ccda1-14cb-4351-9089-d5fa2ad16eee None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81437) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:52:48.505269 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-4f6ccda1-14cb-4351-9089-d5fa2ad16eee None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:52:48.567792 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:52:48.569310 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e07fd279-e7d2-4735-8c42-79d99075220d None None] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:52:48.580231 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:52:48.588336 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 16/31] 10.222.0.50 () {58 vars in 1024 bytes} [Tue Sep 13 15:52:48 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 129 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 13 15:52:48.589530 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f6ccda1-14cb-4351-9089-d5fa2ad16eee None None] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:52:48.639280 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-720910c3-bb25-413f-bb17-4c35b488c382 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:52:48.640314 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-720910c3-bb25-413f-bb17-4c35b488c382 None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:52:48.640823 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-720910c3-bb25-413f-bb17-4c35b488c382 None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:52:48.641388 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-720910c3-bb25-413f-bb17-4c35b488c382 None admin] PATH_INFO: `/v3/roles` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:52:48.642295 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-720910c3-bb25-413f-bb17-4c35b488c382 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81438) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:52:48.642818 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-720910c3-bb25-413f-bb17-4c35b488c382 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:52:48.645735 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-720910c3-bb25-413f-bb17-4c35b488c382 None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:52:48.675539 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 16/32] 10.222.0.50 () {56 vars in 1099 bytes} [Tue Sep 13 15:52:48 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 271 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:52:48.676382 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-720910c3-bb25-413f-bb17-4c35b488c382 None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:52:48.794405 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f6ccda1-14cb-4351-9089-d5fa2ad16eee None None] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:52:48.809979 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-720910c3-bb25-413f-bb17-4c35b488c382 None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:52:48.846378 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a0ace8d-0d98-4c21-8f61-d1e1b5435df4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:52:48.847022 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a0ace8d-0d98-4c21-8f61-d1e1b5435df4 None admin] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:52:48.847306 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a0ace8d-0d98-4c21-8f61-d1e1b5435df4 None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:52:48.847580 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a0ace8d-0d98-4c21-8f61-d1e1b5435df4 None admin] PATH_INFO: `/v3/roles` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:52:48.848375 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a0ace8d-0d98-4c21-8f61-d1e1b5435df4 None admin] RBAC: Authorizing `identity:create_role()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:52:48.849855 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a0ace8d-0d98-4c21-8f61-d1e1b5435df4 None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:52:48.877200 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44ad3475-020a-4901-aade-8c892b566030 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:52:48.879096 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44ad3475-020a-4901-aade-8c892b566030 None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:52:48.879892 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44ad3475-020a-4901-aade-8c892b566030 None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:52:48.880185 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44ad3475-020a-4901-aade-8c892b566030 None admin] PATH_INFO: `/v3/domains` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:52:48.881961 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44ad3475-020a-4901-aade-8c892b566030 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81437) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:52:48.882283 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44ad3475-020a-4901-aade-8c892b566030 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:52:48.886391 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44ad3475-020a-4901-aade-8c892b566030 None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:52:48.928940 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 17/33] 10.222.0.50 () {56 vars in 1109 bytes} [Tue Sep 13 15:52:48 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 340 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:52:48.929812 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44ad3475-020a-4901-aade-8c892b566030 None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:52:48.982828 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:52:49.026928 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 17/34] 10.222.0.50 () {60 vars in 1131 bytes} [Tue Sep 13 15:52:48 2022] POST /identity/v3/roles => generated 219 bytes in 345 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 15:52:49.026928 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a0ace8d-0d98-4c21-8f61-d1e1b5435df4 None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:52:49.068398 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44ad3475-020a-4901-aade-8c892b566030 None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:52:49.069166 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:52:49.118696 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-172538b5-bf6b-491e-b0bb-8bbcb32a4c2d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:52:49.119296 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-172538b5-bf6b-491e-b0bb-8bbcb32a4c2d None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:52:49.120167 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-172538b5-bf6b-491e-b0bb-8bbcb32a4c2d None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:52:49.120460 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-172538b5-bf6b-491e-b0bb-8bbcb32a4c2d None admin] PATH_INFO: `/v3/users/e0a915a0eb3c42c7af97dcc7eed5b42f` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:52:49.122592 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-172538b5-bf6b-491e-b0bb-8bbcb32a4c2d None admin] RBAC: Authorizing `identity:get_user(user_id=e0a915a0eb3c42c7af97dcc7eed5b42f)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:52:49.124122 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-172538b5-bf6b-491e-b0bb-8bbcb32a4c2d None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:52:49.126009 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 18/35] 10.222.0.50 () {56 vars in 1177 bytes} [Tue Sep 13 15:52:48 2022] GET /identity/v3/users/e0a915a0eb3c42c7af97dcc7eed5b42f => generated 247 bytes in 196 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:52:49.126771 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-172538b5-bf6b-491e-b0bb-8bbcb32a4c2d None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:52:49.156515 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a0ace8d-0d98-4c21-8f61-d1e1b5435df4 None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:52:49.199320 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-283af079-94b2-4fe7-83eb-0ce0645cd4e4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:52:49.199953 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-283af079-94b2-4fe7-83eb-0ce0645cd4e4 None admin] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:52:49.200232 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-283af079-94b2-4fe7-83eb-0ce0645cd4e4 None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:52:49.200506 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-283af079-94b2-4fe7-83eb-0ce0645cd4e4 None admin] PATH_INFO: `/v3/roles` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:52:49.201189 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-283af079-94b2-4fe7-83eb-0ce0645cd4e4 None admin] RBAC: Authorizing `identity:create_role()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:52:49.202721 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-283af079-94b2-4fe7-83eb-0ce0645cd4e4 None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:52:49.221769 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 18/36] 10.222.0.50 () {60 vars in 1131 bytes} [Tue Sep 13 15:52:49 2022] POST /identity/v3/roles => generated 225 bytes in 200 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 15:52:49.223268 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8966d50-d6f9-4fda-a88d-e4471404e099 None None] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:52:49.223557 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8966d50-d6f9-4fda-a88d-e4471404e099 None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:52:49.223836 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8966d50-d6f9-4fda-a88d-e4471404e099 None None] PATH_INFO: `/` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:52:49.229638 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 19/37] 10.222.0.50 () {54 vars in 927 bytes} [Tue Sep 13 15:52:49 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:52:49.231144 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8966d50-d6f9-4fda-a88d-e4471404e099 None None] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:52:49.272884 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-172538b5-bf6b-491e-b0bb-8bbcb32a4c2d None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:52:49.293875 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:52:49.319687 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d33a558-7752-40fb-b1d0-53f5662fa59c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:52:49.320707 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d33a558-7752-40fb-b1d0-53f5662fa59c None admin] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:52:49.321196 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d33a558-7752-40fb-b1d0-53f5662fa59c None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:52:49.321785 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d33a558-7752-40fb-b1d0-53f5662fa59c None admin] PATH_INFO: `/v3/projects` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:52:49.322854 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d33a558-7752-40fb-b1d0-53f5662fa59c None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:52:49.324800 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d33a558-7752-40fb-b1d0-53f5662fa59c None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:52:49.376560 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:52:49.393816 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8966d50-d6f9-4fda-a88d-e4471404e099 None None] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:52:49.431763 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abe4b07b-d65b-42c5-b587-4de28508ec2a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:52:49.433140 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abe4b07b-d65b-42c5-b587-4de28508ec2a None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:52:49.433528 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abe4b07b-d65b-42c5-b587-4de28508ec2a None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:52:49.433807 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abe4b07b-d65b-42c5-b587-4de28508ec2a None admin] PATH_INFO: `/v3/domains/default` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:52:49.435320 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abe4b07b-d65b-42c5-b587-4de28508ec2a None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:52:49.436808 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abe4b07b-d65b-42c5-b587-4de28508ec2a None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:52:49.438602 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 20/38] 10.222.0.50 () {56 vars in 1108 bytes} [Tue Sep 13 15:52:49 2022] GET /identity/v3/domains/default => generated 199 bytes in 208 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:52:49.439755 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c51723b-709e-437e-b3d0-25524915a6f5 None None] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:52:49.440042 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c51723b-709e-437e-b3d0-25524915a6f5 None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:52:49.440311 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c51723b-709e-437e-b3d0-25524915a6f5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:52:49.462088 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.auth.core [None req-7c51723b-709e-437e-b3d0-25524915a6f5 None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81438) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:52:49.464087 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-7c51723b-709e-437e-b3d0-25524915a6f5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:52:49.537916 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 21/39] 10.222.0.50 () {58 vars in 1024 bytes} [Tue Sep 13 15:52:49 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 13 15:52:49.539277 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3852dd5-083a-47d2-ae7f-0b7011bf9fef None None] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:52:49.539577 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3852dd5-083a-47d2-ae7f-0b7011bf9fef None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:52:49.539858 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3852dd5-083a-47d2-ae7f-0b7011bf9fef None None] PATH_INFO: `/` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:52:49.543006 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 22/40] 10.222.0.50 () {54 vars in 927 bytes} [Tue Sep 13 15:52:49 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:52:49.543006 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3852dd5-083a-47d2-ae7f-0b7011bf9fef None None] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:52:49.581290 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 19/41] 10.222.0.50 () {60 vars in 1141 bytes} [Tue Sep 13 15:52:49 2022] POST /identity/v3/projects => generated 301 bytes in 455 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 15:52:49.582740 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d094aa8e-9b5f-45be-a736-caad3bd89253 None None] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:52:49.583057 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d094aa8e-9b5f-45be-a736-caad3bd89253 None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:52:49.583341 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d094aa8e-9b5f-45be-a736-caad3bd89253 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:52:49.608399 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.auth.core [None req-d094aa8e-9b5f-45be-a736-caad3bd89253 None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81437) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:52:49.610127 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-d094aa8e-9b5f-45be-a736-caad3bd89253 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:52:49.658349 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:52:49.678741 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3852dd5-083a-47d2-ae7f-0b7011bf9fef None None] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:52:49.717935 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 20/42] 10.222.0.50 () {58 vars in 1024 bytes} [Tue Sep 13 15:52:49 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 136 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 13 15:52:49.730928 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1dfe5dd7-13c9-44fc-8a75-38377e5bed19 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:52:49.730928 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dfe5dd7-13c9-44fc-8a75-38377e5bed19 None admin] REQUEST_METHOD: `PUT` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:52:49.730928 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dfe5dd7-13c9-44fc-8a75-38377e5bed19 None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:52:49.730928 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dfe5dd7-13c9-44fc-8a75-38377e5bed19 None admin] PATH_INFO: `/v3/domains/default/users/e0a915a0eb3c42c7af97dcc7eed5b42f/roles/031d4e1a23e34f71858fc4a6dfbd4ccb` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:52:49.731345 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1dfe5dd7-13c9-44fc-8a75-38377e5bed19 None admin] RBAC: Authorizing `identity:create_grant(domain_id=default, user_id=e0a915a0eb3c42c7af97dcc7eed5b42f, role_id=031d4e1a23e34f71858fc4a6dfbd4ccb)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:52:49.731345 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1dfe5dd7-13c9-44fc-8a75-38377e5bed19 None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:52:49.752403 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 23/43] 10.222.0.50 () {58 vars in 1361 bytes} [Tue Sep 13 15:52:49 2022] PUT /identity/v3/domains/default/users/e0a915a0eb3c42c7af97dcc7eed5b42f/roles/031d4e1a23e34f71858fc4a6dfbd4ccb => generated 0 bytes in 210 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:52:49.821137 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d094aa8e-9b5f-45be-a736-caad3bd89253 None None] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:52:49.927185 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:52:49.936886 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d094aa8e-9b5f-45be-a736-caad3bd89253 None None] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:52:49.971991 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d9be146-9ad3-4669-b436-8df1f0a218df None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:52:49.972605 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d9be146-9ad3-4669-b436-8df1f0a218df None admin] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:52:49.972885 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d9be146-9ad3-4669-b436-8df1f0a218df None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:52:49.973158 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d9be146-9ad3-4669-b436-8df1f0a218df None admin] PATH_INFO: `/v3/roles` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:52:49.973837 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d9be146-9ad3-4669-b436-8df1f0a218df None admin] RBAC: Authorizing `identity:create_role()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:52:49.975379 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d9be146-9ad3-4669-b436-8df1f0a218df None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:52:49.987304 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1dfe5dd7-13c9-44fc-8a75-38377e5bed19 None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:52:49.991116 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 21/44] 10.222.0.50 () {60 vars in 1131 bytes} [Tue Sep 13 15:52:49 2022] POST /identity/v3/roles => generated 223 bytes in 171 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 15:52:50.103432 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1dfe5dd7-13c9-44fc-8a75-38377e5bed19 None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:52:50.141966 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36223688-10d6-4be1-981e-348c4f9dfd3a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:52:50.142626 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36223688-10d6-4be1-981e-348c4f9dfd3a None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:52:50.142929 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36223688-10d6-4be1-981e-348c4f9dfd3a None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:52:50.143220 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36223688-10d6-4be1-981e-348c4f9dfd3a None admin] PATH_INFO: `/v3/domains/default` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:52:50.144384 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36223688-10d6-4be1-981e-348c4f9dfd3a None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:52:50.145867 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36223688-10d6-4be1-981e-348c4f9dfd3a None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:52:50.147756 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 24/45] 10.222.0.50 () {56 vars in 1108 bytes} [Tue Sep 13 15:52:49 2022] GET /identity/v3/domains/default => generated 199 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:52:50.153255 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d9be146-9ad3-4669-b436-8df1f0a218df None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:52:50.262193 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d9be146-9ad3-4669-b436-8df1f0a218df None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:52:50.295477 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09517ee9-05a6-4790-92b5-46b7b261d678 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:52:50.296088 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09517ee9-05a6-4790-92b5-46b7b261d678 None admin] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:52:50.296372 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09517ee9-05a6-4790-92b5-46b7b261d678 None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:52:50.296649 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09517ee9-05a6-4790-92b5-46b7b261d678 None admin] PATH_INFO: `/v3/projects` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:52:50.297310 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09517ee9-05a6-4790-92b5-46b7b261d678 None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:52:50.298770 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09517ee9-05a6-4790-92b5-46b7b261d678 None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:52:50.325518 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 22/46] 10.222.0.50 () {60 vars in 1141 bytes} [Tue Sep 13 15:52:50 2022] POST /identity/v3/projects => generated 312 bytes in 173 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 15:52:51.958420 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2c289a3-b389-4b49-b172-037a8575ded5 None None] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:52:51.959083 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2c289a3-b389-4b49-b172-037a8575ded5 None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:52:51.959083 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2c289a3-b389-4b49-b172-037a8575ded5 None None] PATH_INFO: `/` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:52:51.960355 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 25/47] 10.222.0.50 () {54 vars in 927 bytes} [Tue Sep 13 15:52:51 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:52:51.966078 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdcad195-7c6b-4141-a4c7-2020e461ce85 None None] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:52:51.966463 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdcad195-7c6b-4141-a4c7-2020e461ce85 None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:52:51.966754 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdcad195-7c6b-4141-a4c7-2020e461ce85 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:52:51.984509 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.auth.core [None req-cdcad195-7c6b-4141-a4c7-2020e461ce85 None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81437) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:52:51.985678 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-cdcad195-7c6b-4141-a4c7-2020e461ce85 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:52:51.987641 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:52:52.052497 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 23/48] 10.222.0.50 () {58 vars in 1024 bytes} [Tue Sep 13 15:52:51 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 13 15:52:52.325542 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2c289a3-b389-4b49-b172-037a8575ded5 None None] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:52:52.342169 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:52:52.431460 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2c289a3-b389-4b49-b172-037a8575ded5 None None] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:52:52.464224 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8e5350d-e767-4cdd-bc77-6e9522aae7c9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:52:52.464853 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8e5350d-e767-4cdd-bc77-6e9522aae7c9 None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:52:52.465135 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8e5350d-e767-4cdd-bc77-6e9522aae7c9 None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:52:52.465410 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8e5350d-e767-4cdd-bc77-6e9522aae7c9 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:52:52.475781 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8e5350d-e767-4cdd-bc77-6e9522aae7c9 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 13 15:52:52.479532 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdfb3e75-eab6-4759-9e9c-152f2fe2df98 None None] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:52:52.479851 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdfb3e75-eab6-4759-9e9c-152f2fe2df98 None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:52:52.480129 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdfb3e75-eab6-4759-9e9c-152f2fe2df98 None None] PATH_INFO: `/` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:52:52.481503 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 24/49] 10.222.0.50 () {54 vars in 927 bytes} [Tue Sep 13 15:52:52 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 13 15:52:52.481602 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application [None req-b8e5350d-e767-4cdd-bc77-6e9522aae7c9 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 13 15:52:52.481602 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:52:52.481602 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:52:52.481602 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:52:52.481602 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:52:52.481602 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:52:52.481602 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:52:52.481602 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:52:52.481602 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:52:52.481602 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:52:52.481602 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:52:52.481602 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:52:52.481602 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 13 15:52:52.481602 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 13 15:52:52.481602 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 13 15:52:52.481602 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 13 15:52:52.482403 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 13 15:52:52.482403 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 13 15:52:52.482403 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 13 15:52:52.482403 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 13 15:52:52.482403 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:52:52.482403 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:52:52.482403 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:52:52.482403 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:52:52.482403 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:52:52.482403 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:52:52.482403 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:52:52.482403 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application with Lock( Sep 13 15:52:52.482403 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:52:52.482403 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:52:52.482403 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:52:52.482403 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:52:52.483209 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:52:52.483209 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:52:52.483209 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:52:52.483209 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:52:52.483209 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 13 15:52:52.483209 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 13 15:52:52.483209 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 13 15:52:52.483209 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 13 15:52:52.483209 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 13 15:52:52.483209 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 13 15:52:52.483209 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 13 15:52:52.483209 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Sep 13 15:52:52.483209 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 26/50] 10.222.0.50 () {56 vars in 1096 bytes} [Tue Sep 13 15:52:52 2022] GET /identity/v3/roles/admin => generated 83 bytes in 158 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:52:52.486477 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-599728c1-2d07-405d-beca-e1cb546cfc26 None None] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:52:52.486762 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-599728c1-2d07-405d-beca-e1cb546cfc26 None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:52:52.487078 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-599728c1-2d07-405d-beca-e1cb546cfc26 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:52:52.488225 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8e5350d-e767-4cdd-bc77-6e9522aae7c9 None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:52:52.505527 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.auth.core [None req-599728c1-2d07-405d-beca-e1cb546cfc26 None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81437) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:52:52.506727 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-599728c1-2d07-405d-beca-e1cb546cfc26 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:52:52.564375 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:52:52.573472 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 25/51] 10.222.0.50 () {58 vars in 1024 bytes} [Tue Sep 13 15:52:52 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 13 15:52:52.590052 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8e5350d-e767-4cdd-bc77-6e9522aae7c9 None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:52:52.623268 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9491a9f7-d117-46ad-b73c-5210e04b63eb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:52:52.623883 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9491a9f7-d117-46ad-b73c-5210e04b63eb None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:52:52.624168 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9491a9f7-d117-46ad-b73c-5210e04b63eb None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:52:52.624447 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9491a9f7-d117-46ad-b73c-5210e04b63eb None admin] PATH_INFO: `/v3/roles` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:52:52.625180 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9491a9f7-d117-46ad-b73c-5210e04b63eb None admin] RBAC: Adding query filter params (name=admin) {{(pid=81438) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:52:52.625490 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9491a9f7-d117-46ad-b73c-5210e04b63eb None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:52:52.626879 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9491a9f7-d117-46ad-b73c-5210e04b63eb None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:52:52.635169 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 27/52] 10.222.0.50 () {56 vars in 1099 bytes} [Tue Sep 13 15:52:52 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 13 15:52:52.640855 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-599728c1-2d07-405d-beca-e1cb546cfc26 None None] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:52:52.707216 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:52:52.749097 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-599728c1-2d07-405d-beca-e1cb546cfc26 None None] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:52:52.790293 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-417c8fe6-3e0a-4a88-a117-d9877910ea09 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:52:52.790943 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-417c8fe6-3e0a-4a88-a117-d9877910ea09 None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:52:52.791228 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-417c8fe6-3e0a-4a88-a117-d9877910ea09 None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:52:52.792133 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-417c8fe6-3e0a-4a88-a117-d9877910ea09 None admin] PATH_INFO: `/v3/users/e0a915a0eb3c42c7af97dcc7eed5b42f` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:52:52.793251 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-417c8fe6-3e0a-4a88-a117-d9877910ea09 None admin] RBAC: Authorizing `identity:get_user(user_id=e0a915a0eb3c42c7af97dcc7eed5b42f)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:52:52.794701 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-417c8fe6-3e0a-4a88-a117-d9877910ea09 None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:52:52.796308 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 26/53] 10.222.0.50 () {56 vars in 1177 bytes} [Tue Sep 13 15:52:52 2022] GET /identity/v3/users/e0a915a0eb3c42c7af97dcc7eed5b42f => generated 247 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:52:52.801709 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9491a9f7-d117-46ad-b73c-5210e04b63eb None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:52:52.847807 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-417c8fe6-3e0a-4a88-a117-d9877910ea09 None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:52:52.901541 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9491a9f7-d117-46ad-b73c-5210e04b63eb None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:52:52.950480 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-417c8fe6-3e0a-4a88-a117-d9877910ea09 None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:52:52.986779 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20d1c869-b9c6-4abd-8bc9-0f65496c49b6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:52:52.987491 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20d1c869-b9c6-4abd-8bc9-0f65496c49b6 None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:52:52.987728 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20d1c869-b9c6-4abd-8bc9-0f65496c49b6 None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:52:52.988002 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20d1c869-b9c6-4abd-8bc9-0f65496c49b6 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:52:52.989185 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20d1c869-b9c6-4abd-8bc9-0f65496c49b6 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:52:52.990688 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20d1c869-b9c6-4abd-8bc9-0f65496c49b6 None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:52:52.992366 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 27/54] 10.222.0.50 () {56 vars in 1108 bytes} [Tue Sep 13 15:52:52 2022] GET /identity/v3/domains/default => generated 199 bytes in 145 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 13 15:52:52.998715 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20d1c869-b9c6-4abd-8bc9-0f65496c49b6 None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:52:53.002580 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:52:53.039157 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2709a5f2-ba6c-4803-8621-9ac9128d95b7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:52:53.039860 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2709a5f2-ba6c-4803-8621-9ac9128d95b7 None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:52:53.040504 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2709a5f2-ba6c-4803-8621-9ac9128d95b7 None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:52:53.040791 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2709a5f2-ba6c-4803-8621-9ac9128d95b7 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:52:53.041967 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2709a5f2-ba6c-4803-8621-9ac9128d95b7 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:52:53.044139 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2709a5f2-ba6c-4803-8621-9ac9128d95b7 None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:52:53.045742 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 28/55] 10.222.0.50 () {56 vars in 1108 bytes} [Tue Sep 13 15:52:52 2022] GET /identity/v3/domains/default => generated 199 bytes in 245 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:52:53.051510 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2709a5f2-ba6c-4803-8621-9ac9128d95b7 None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:52:53.055049 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:52:53.101624 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20d1c869-b9c6-4abd-8bc9-0f65496c49b6 None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:52:53.135178 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76a36701-be00-4905-aaf7-5bc9fef896da None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:52:53.135860 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76a36701-be00-4905-aaf7-5bc9fef896da None admin] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:52:53.136148 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76a36701-be00-4905-aaf7-5bc9fef896da None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:52:53.136424 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76a36701-be00-4905-aaf7-5bc9fef896da None admin] PATH_INFO: `/v3/projects` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:52:53.137069 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76a36701-be00-4905-aaf7-5bc9fef896da None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:52:53.138514 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76a36701-be00-4905-aaf7-5bc9fef896da None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:52:53.153949 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2709a5f2-ba6c-4803-8621-9ac9128d95b7 None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:52:53.164204 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 28/56] 10.222.0.50 () {60 vars in 1140 bytes} [Tue Sep 13 15:52:52 2022] POST /identity/v3/projects => generated 298 bytes in 166 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 13 15:52:53.191214 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d23884d0-1af4-4f57-89e7-532dcd473b63 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:52:53.191851 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d23884d0-1af4-4f57-89e7-532dcd473b63 None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:52:53.192134 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d23884d0-1af4-4f57-89e7-532dcd473b63 None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:52:53.192411 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d23884d0-1af4-4f57-89e7-532dcd473b63 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:52:53.193159 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d23884d0-1af4-4f57-89e7-532dcd473b63 None admin] RBAC: Adding query filter params (role.id=031d4e1a23e34f71858fc4a6dfbd4ccb, scope.domain.id=default, user.id=e0a915a0eb3c42c7af97dcc7eed5b42f) {{(pid=81438) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:52:53.193473 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d23884d0-1af4-4f57-89e7-532dcd473b63 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:52:53.194993 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d23884d0-1af4-4f57-89e7-532dcd473b63 None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:52:53.204737 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 29/57] 10.222.0.50 () {56 vars in 1322 bytes} [Tue Sep 13 15:52:53 2022] GET /identity/v3/role_assignments?user.id=e0a915a0eb3c42c7af97dcc7eed5b42f&scope.domain.id=default&role.id=031d4e1a23e34f71858fc4a6dfbd4ccb => generated 530 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 13 15:52:55.321409 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4046e0f-bc07-4b6f-8b7b-525b5bb66540 None None] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:52:55.321904 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4046e0f-bc07-4b6f-8b7b-525b5bb66540 None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:52:55.321904 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4046e0f-bc07-4b6f-8b7b-525b5bb66540 None None] PATH_INFO: `/` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:52:55.324131 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 29/58] 10.222.0.50 () {54 vars in 927 bytes} [Tue Sep 13 15:52:55 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:52:55.329197 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69bcd003-5659-436f-bf12-29888db2d3ec None None] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:52:55.329624 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69bcd003-5659-436f-bf12-29888db2d3ec None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:52:55.329758 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69bcd003-5659-436f-bf12-29888db2d3ec None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:52:55.348224 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.auth.core [None req-69bcd003-5659-436f-bf12-29888db2d3ec None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81438) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:52:55.349420 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-69bcd003-5659-436f-bf12-29888db2d3ec None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:52:55.408592 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:52:55.413958 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 30/59] 10.222.0.50 () {58 vars in 1024 bytes} [Tue Sep 13 15:52:55 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 13 15:52:55.683727 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4046e0f-bc07-4b6f-8b7b-525b5bb66540 None None] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:52:55.740443 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:52:55.785145 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4046e0f-bc07-4b6f-8b7b-525b5bb66540 None None] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:52:55.818401 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98c7fb7e-585d-4938-8cf9-1b431dc86e94 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:52:55.819074 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98c7fb7e-585d-4938-8cf9-1b431dc86e94 None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:52:55.819355 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98c7fb7e-585d-4938-8cf9-1b431dc86e94 None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:52:55.819629 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98c7fb7e-585d-4938-8cf9-1b431dc86e94 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:52:55.821437 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98c7fb7e-585d-4938-8cf9-1b431dc86e94 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:52:55.823510 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98c7fb7e-585d-4938-8cf9-1b431dc86e94 None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:52:55.825070 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 30/60] 10.222.0.50 () {56 vars in 1108 bytes} [Tue Sep 13 15:52:55 2022] GET /identity/v3/domains/default => generated 199 bytes in 142 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 13 15:52:55.830692 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69bcd003-5659-436f-bf12-29888db2d3ec None None] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:52:55.926170 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69bcd003-5659-436f-bf12-29888db2d3ec None None] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:52:55.959859 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8115f83a-ae51-40f5-bfab-608921043e5a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:52:55.960403 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8115f83a-ae51-40f5-bfab-608921043e5a None admin] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:52:55.960688 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8115f83a-ae51-40f5-bfab-608921043e5a None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:52:55.960964 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8115f83a-ae51-40f5-bfab-608921043e5a None admin] PATH_INFO: `/v3/users` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:52:55.961612 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8115f83a-ae51-40f5-bfab-608921043e5a None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:52:55.963069 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8115f83a-ae51-40f5-bfab-608921043e5a None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:52:55.988611 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 31/61] 10.222.0.50 () {60 vars in 1132 bytes} [Tue Sep 13 15:52:55 2022] POST /identity/v3/users => generated 275 bytes in 158 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 15:52:59.281313 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f086e619-dacf-47c4-85bd-f3b26295e761 None None] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:52:59.282331 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f086e619-dacf-47c4-85bd-f3b26295e761 None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:52:59.282824 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f086e619-dacf-47c4-85bd-f3b26295e761 None None] PATH_INFO: `/` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:52:59.284669 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 31/62] 10.222.0.50 () {54 vars in 927 bytes} [Tue Sep 13 15:52:59 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:52:59.290253 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b820105f-76bf-4e52-9df1-2a886e6e8049 None None] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:52:59.291121 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b820105f-76bf-4e52-9df1-2a886e6e8049 None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:52:59.292406 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b820105f-76bf-4e52-9df1-2a886e6e8049 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:52:59.318021 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.auth.core [None req-b820105f-76bf-4e52-9df1-2a886e6e8049 None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81438) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:52:59.319587 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-b820105f-76bf-4e52-9df1-2a886e6e8049 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:52:59.378880 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:52:59.405311 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 32/63] 10.222.0.50 () {58 vars in 1024 bytes} [Tue Sep 13 15:52:59 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 116 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 13 15:52:59.545053 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d0eb35d-74d8-42ba-9779-e2416f7a7c91 None None] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:52:59.545569 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d0eb35d-74d8-42ba-9779-e2416f7a7c91 None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:52:59.546021 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d0eb35d-74d8-42ba-9779-e2416f7a7c91 None None] PATH_INFO: `/` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:52:59.547760 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 32/64] 10.222.0.50 () {54 vars in 927 bytes} [Tue Sep 13 15:52:59 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:52:59.561574 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5e0271f-5fe4-4d30-87db-6aa3880c9075 None None] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:52:59.561867 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5e0271f-5fe4-4d30-87db-6aa3880c9075 None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:52:59.562150 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5e0271f-5fe4-4d30-87db-6aa3880c9075 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:52:59.588001 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.auth.core [None req-e5e0271f-5fe4-4d30-87db-6aa3880c9075 None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81438) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:52:59.589279 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-e5e0271f-5fe4-4d30-87db-6aa3880c9075 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:52:59.629765 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4ac65cb-2687-4d46-ab93-021eace53d66 None None] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:52:59.630376 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4ac65cb-2687-4d46-ab93-021eace53d66 None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:52:59.630858 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4ac65cb-2687-4d46-ab93-021eace53d66 None None] PATH_INFO: `/` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:52:59.632685 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 33/65] 10.222.0.50 () {54 vars in 927 bytes} [Tue Sep 13 15:52:59 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 13 15:52:59.638347 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2597281f-ae32-4066-9497-7e59f06766e0 None None] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:52:59.638924 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2597281f-ae32-4066-9497-7e59f06766e0 None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:52:59.639411 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2597281f-ae32-4066-9497-7e59f06766e0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:52:59.665583 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.auth.core [None req-2597281f-ae32-4066-9497-7e59f06766e0 None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81437) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:52:59.668060 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-2597281f-ae32-4066-9497-7e59f06766e0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:52:59.689550 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 33/66] 10.222.0.50 () {58 vars in 1024 bytes} [Tue Sep 13 15:52:59 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 129 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 13 15:52:59.741052 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 34/67] 10.222.0.50 () {58 vars in 1024 bytes} [Tue Sep 13 15:52:59 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 13 15:52:59.801284 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5e0271f-5fe4-4d30-87db-6aa3880c9075 None None] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:52:59.909985 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-285fdd1a-7cfb-4e23-a933-64a191372216 None None] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:52:59.910589 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-285fdd1a-7cfb-4e23-a933-64a191372216 None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:52:59.911244 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-285fdd1a-7cfb-4e23-a933-64a191372216 None None] PATH_INFO: `/` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:52:59.912993 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 35/68] 10.222.0.50 () {54 vars in 927 bytes} [Tue Sep 13 15:52:59 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 13 15:52:59.919287 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cd7f987-a411-481a-a215-8d44bca71c17 None None] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:52:59.919820 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cd7f987-a411-481a-a215-8d44bca71c17 None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:52:59.920296 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cd7f987-a411-481a-a215-8d44bca71c17 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:52:59.936314 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5e0271f-5fe4-4d30-87db-6aa3880c9075 None None] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:52:59.940448 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.auth.core [None req-5cd7f987-a411-481a-a215-8d44bca71c17 None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81437) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:52:59.941853 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-5cd7f987-a411-481a-a215-8d44bca71c17 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:52:59.951553 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:52:59.974833 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-961f8c29-ce12-4cde-ab43-0718dde7c64d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:52:59.975608 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-961f8c29-ce12-4cde-ab43-0718dde7c64d None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:52:59.975896 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-961f8c29-ce12-4cde-ab43-0718dde7c64d None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:52:59.976177 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-961f8c29-ce12-4cde-ab43-0718dde7c64d None admin] PATH_INFO: `/v3/roles/member` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:52:59.987969 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-961f8c29-ce12-4cde-ab43-0718dde7c64d None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 13 15:52:59.993241 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application [None req-961f8c29-ce12-4cde-ab43-0718dde7c64d None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 13 15:52:59.993241 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:52:59.993241 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:52:59.993241 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:52:59.993241 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:52:59.993241 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:52:59.993241 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:52:59.993241 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:52:59.993241 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:52:59.993241 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:52:59.993241 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:52:59.993241 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:52:59.993241 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 13 15:52:59.993241 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 13 15:52:59.993241 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 13 15:52:59.993241 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 13 15:52:59.994105 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 13 15:52:59.994105 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 13 15:52:59.994105 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 13 15:52:59.994105 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 13 15:52:59.994105 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:52:59.994105 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:52:59.994105 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:52:59.994105 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:52:59.994105 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:52:59.994105 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:52:59.994105 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:52:59.994105 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application with Lock( Sep 13 15:52:59.994105 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:52:59.994105 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:52:59.994105 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:52:59.994105 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:52:59.995628 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:52:59.995628 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:52:59.995628 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:52:59.995628 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:52:59.995628 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 13 15:52:59.995628 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 13 15:52:59.995628 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 13 15:52:59.995628 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 13 15:52:59.995628 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 13 15:52:59.995628 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 13 15:52:59.995628 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 13 15:52:59.995628 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Sep 13 15:52:59.995628 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 34/69] 10.222.0.50 () {56 vars in 1099 bytes} [Tue Sep 13 15:52:59 2022] GET /identity/v3/roles/member => generated 84 bytes in 194 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:52:59.996759 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-546a1a3f-d922-428f-9b4a-518206804918 None None] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:52:59.997056 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-546a1a3f-d922-428f-9b4a-518206804918 None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:52:59.997336 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-546a1a3f-d922-428f-9b4a-518206804918 None None] PATH_INFO: `/` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:52:59.998876 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 35/70] 10.222.0.50 () {54 vars in 927 bytes} [Tue Sep 13 15:52:59 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:53:00.003588 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-546a1a3f-d922-428f-9b4a-518206804918 None None] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:53:00.005600 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:53:00.026831 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 36/71] 10.222.0.50 () {58 vars in 1024 bytes} [Tue Sep 13 15:52:59 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 13 15:53:00.028302 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b0b0e9b-2f26-416a-a92b-b26fed3ecfa0 None None] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:00.028890 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b0b0e9b-2f26-416a-a92b-b26fed3ecfa0 None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:00.029301 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b0b0e9b-2f26-416a-a92b-b26fed3ecfa0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:00.071900 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.auth.core [None req-3b0b0e9b-2f26-416a-a92b-b26fed3ecfa0 None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81437) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:53:00.073414 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-3b0b0e9b-2f26-416a-a92b-b26fed3ecfa0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:53:00.149152 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-546a1a3f-d922-428f-9b4a-518206804918 None None] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:53:00.155816 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 37/72] 10.222.0.50 () {58 vars in 1024 bytes} [Tue Sep 13 15:53:00 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 128 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 13 15:53:00.156589 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b0b0e9b-2f26-416a-a92b-b26fed3ecfa0 None None] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:53:00.186562 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d382e942-a8cb-4447-b40b-9987fb73d636 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:53:00.187228 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d382e942-a8cb-4447-b40b-9987fb73d636 None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:00.187583 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d382e942-a8cb-4447-b40b-9987fb73d636 None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:00.187930 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d382e942-a8cb-4447-b40b-9987fb73d636 None admin] PATH_INFO: `/v3/roles` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:00.188670 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d382e942-a8cb-4447-b40b-9987fb73d636 None admin] RBAC: Adding query filter params (name=member) {{(pid=81438) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:53:00.189045 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d382e942-a8cb-4447-b40b-9987fb73d636 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:53:00.190506 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d382e942-a8cb-4447-b40b-9987fb73d636 None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:53:00.201110 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 36/73] 10.222.0.50 () {56 vars in 1101 bytes} [Tue Sep 13 15:53:00 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 198 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:53:00.202074 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d382e942-a8cb-4447-b40b-9987fb73d636 None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:53:00.239619 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:53:00.296673 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b0b0e9b-2f26-416a-a92b-b26fed3ecfa0 None None] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:53:00.316515 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d382e942-a8cb-4447-b40b-9987fb73d636 None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:53:00.316992 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:53:00.341707 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccd2e2dc-6e93-47b3-bee1-3922e02131d2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:53:00.342348 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccd2e2dc-6e93-47b3-bee1-3922e02131d2 None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:00.342632 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccd2e2dc-6e93-47b3-bee1-3922e02131d2 None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:00.343106 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccd2e2dc-6e93-47b3-bee1-3922e02131d2 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:00.353509 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ccd2e2dc-6e93-47b3-bee1-3922e02131d2 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 13 15:53:00.355208 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e45f2619-d54e-4265-a76e-e79097a4f5f5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:53:00.356128 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e45f2619-d54e-4265-a76e-e79097a4f5f5 None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:00.356629 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e45f2619-d54e-4265-a76e-e79097a4f5f5 None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:00.357117 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e45f2619-d54e-4265-a76e-e79097a4f5f5 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:00.359055 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application [None req-ccd2e2dc-6e93-47b3-bee1-3922e02131d2 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 13 15:53:00.359055 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:53:00.359055 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:53:00.359055 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:53:00.359055 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:53:00.359055 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:53:00.359055 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:53:00.359055 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:53:00.359055 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:53:00.359055 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:53:00.359055 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:53:00.359055 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:53:00.359055 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 13 15:53:00.359055 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 13 15:53:00.359055 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 13 15:53:00.359055 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 13 15:53:00.360134 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 13 15:53:00.360134 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 13 15:53:00.360134 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 13 15:53:00.360134 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 13 15:53:00.360134 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:53:00.360134 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:53:00.360134 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:53:00.360134 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:53:00.360134 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:53:00.360134 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:53:00.360134 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:53:00.360134 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application with Lock( Sep 13 15:53:00.360134 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:53:00.360134 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:53:00.360134 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:53:00.360134 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:53:00.361088 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:53:00.361088 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:53:00.361088 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:53:00.361088 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:53:00.361088 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 13 15:53:00.361088 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 13 15:53:00.361088 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 13 15:53:00.361088 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 13 15:53:00.361088 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 13 15:53:00.361088 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 13 15:53:00.361088 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 13 15:53:00.361088 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application Sep 13 15:53:00.361088 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 38/74] 10.222.0.50 () {56 vars in 1114 bytes} [Tue Sep 13 15:53:00 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 204 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:53:00.361867 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccd2e2dc-6e93-47b3-bee1-3922e02131d2 None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:53:00.374539 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e45f2619-d54e-4265-a76e-e79097a4f5f5 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 13 15:53:00.379932 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application [None req-e45f2619-d54e-4265-a76e-e79097a4f5f5 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 13 15:53:00.379932 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:53:00.379932 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:53:00.379932 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:53:00.379932 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:53:00.379932 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:53:00.379932 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:53:00.379932 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:53:00.379932 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:53:00.379932 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:53:00.379932 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:53:00.379932 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:53:00.379932 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 13 15:53:00.379932 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 13 15:53:00.379932 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 13 15:53:00.379932 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 13 15:53:00.380793 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 13 15:53:00.380793 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 13 15:53:00.380793 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 13 15:53:00.380793 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 13 15:53:00.380793 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:53:00.380793 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:53:00.380793 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:53:00.380793 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:53:00.380793 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:53:00.380793 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:53:00.380793 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:53:00.380793 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application with Lock( Sep 13 15:53:00.380793 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:53:00.380793 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:53:00.380793 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:53:00.380793 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:53:00.381561 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:53:00.381561 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:53:00.381561 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:53:00.381561 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:53:00.381561 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 13 15:53:00.381561 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 13 15:53:00.381561 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 13 15:53:00.381561 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 13 15:53:00.381561 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 13 15:53:00.381561 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 13 15:53:00.381561 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 13 15:53:00.381561 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Sep 13 15:53:00.381561 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 37/75] 10.222.0.50 () {56 vars in 1096 bytes} [Tue Sep 13 15:53:00 2022] GET /identity/v3/roles/admin => generated 83 bytes in 180 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:53:00.382189 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e45f2619-d54e-4265-a76e-e79097a4f5f5 None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:53:00.477897 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccd2e2dc-6e93-47b3-bee1-3922e02131d2 None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:53:00.484625 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:53:00.491358 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e45f2619-d54e-4265-a76e-e79097a4f5f5 None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:53:00.512440 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-709de504-38bd-4c53-8aa1-1d1606696228 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:53:00.513105 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-709de504-38bd-4c53-8aa1-1d1606696228 None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:00.513393 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-709de504-38bd-4c53-8aa1-1d1606696228 None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:00.513674 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-709de504-38bd-4c53-8aa1-1d1606696228 None admin] PATH_INFO: `/v3/users/5c4340db782249918182a2e21023cc55` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:00.525873 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f63810f-58c8-4224-909d-a838b3bbeac1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:53:00.526557 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f63810f-58c8-4224-909d-a838b3bbeac1 None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:00.526682 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-709de504-38bd-4c53-8aa1-1d1606696228 None admin] RBAC: Authorizing `identity:get_user(user_id=5c4340db782249918182a2e21023cc55)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:53:00.526837 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f63810f-58c8-4224-909d-a838b3bbeac1 None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:00.527142 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f63810f-58c8-4224-909d-a838b3bbeac1 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:00.528069 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-709de504-38bd-4c53-8aa1-1d1606696228 None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:53:00.529686 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 39/76] 10.222.0.50 () {56 vars in 1177 bytes} [Tue Sep 13 15:53:00 2022] GET /identity/v3/users/5c4340db782249918182a2e21023cc55 => generated 275 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:53:00.530471 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-709de504-38bd-4c53-8aa1-1d1606696228 None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:53:00.538527 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f63810f-58c8-4224-909d-a838b3bbeac1 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 13 15:53:00.543644 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application [None req-6f63810f-58c8-4224-909d-a838b3bbeac1 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 13 15:53:00.543644 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:53:00.543644 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:53:00.543644 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:53:00.543644 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:53:00.543644 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:53:00.543644 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:53:00.543644 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:53:00.543644 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:53:00.543644 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:53:00.543644 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:53:00.543644 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:53:00.543644 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 13 15:53:00.543644 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 13 15:53:00.543644 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 13 15:53:00.543644 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 13 15:53:00.544516 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 13 15:53:00.544516 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 13 15:53:00.544516 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 13 15:53:00.544516 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 13 15:53:00.544516 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:53:00.544516 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:53:00.544516 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:53:00.544516 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:53:00.544516 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:53:00.544516 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:53:00.544516 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:53:00.544516 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application with Lock( Sep 13 15:53:00.544516 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:53:00.544516 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:53:00.544516 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:53:00.544516 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:53:00.545437 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:53:00.545437 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:53:00.545437 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:53:00.545437 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:53:00.545437 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 13 15:53:00.545437 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 13 15:53:00.545437 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 13 15:53:00.545437 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 13 15:53:00.545437 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 13 15:53:00.545437 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 13 15:53:00.545437 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 13 15:53:00.545437 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Sep 13 15:53:00.545437 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 38/77] 10.222.0.50 () {56 vars in 1099 bytes} [Tue Sep 13 15:53:00 2022] GET /identity/v3/roles/member => generated 84 bytes in 163 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:53:00.547708 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f63810f-58c8-4224-909d-a838b3bbeac1 None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:53:00.563791 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:53:00.633647 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-709de504-38bd-4c53-8aa1-1d1606696228 None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:53:00.645878 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f63810f-58c8-4224-909d-a838b3bbeac1 None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:53:00.671408 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afc3c8a8-a1d6-4e2c-a707-2b15ba209647 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:53:00.672022 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afc3c8a8-a1d6-4e2c-a707-2b15ba209647 None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:00.672303 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afc3c8a8-a1d6-4e2c-a707-2b15ba209647 None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:00.672577 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afc3c8a8-a1d6-4e2c-a707-2b15ba209647 None admin] PATH_INFO: `/v3/roles` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:00.673239 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afc3c8a8-a1d6-4e2c-a707-2b15ba209647 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81437) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:53:00.673544 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afc3c8a8-a1d6-4e2c-a707-2b15ba209647 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:53:00.674944 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afc3c8a8-a1d6-4e2c-a707-2b15ba209647 None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:53:00.683070 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63ea3ec5-cd81-4e79-9f74-d2736d049752 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:53:00.683630 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 40/78] 10.222.0.50 () {56 vars in 1111 bytes} [Tue Sep 13 15:53:00 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:53:00.683753 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63ea3ec5-cd81-4e79-9f74-d2736d049752 None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:00.683947 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63ea3ec5-cd81-4e79-9f74-d2736d049752 None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:00.684223 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63ea3ec5-cd81-4e79-9f74-d2736d049752 None admin] PATH_INFO: `/v3/roles` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:00.684410 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afc3c8a8-a1d6-4e2c-a707-2b15ba209647 None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:53:00.684959 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63ea3ec5-cd81-4e79-9f74-d2736d049752 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81438) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:53:00.685269 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63ea3ec5-cd81-4e79-9f74-d2736d049752 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:53:00.686635 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63ea3ec5-cd81-4e79-9f74-d2736d049752 None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:53:00.694944 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 39/79] 10.222.0.50 () {56 vars in 1099 bytes} [Tue Sep 13 15:53:00 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:53:00.699894 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63ea3ec5-cd81-4e79-9f74-d2736d049752 None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:53:00.718111 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:53:00.778200 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:53:00.785296 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afc3c8a8-a1d6-4e2c-a707-2b15ba209647 None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:53:00.800289 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63ea3ec5-cd81-4e79-9f74-d2736d049752 None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:53:00.819067 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24e29a8c-1cc4-4abf-b1ff-6cd2ebf1273d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:53:00.819692 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24e29a8c-1cc4-4abf-b1ff-6cd2ebf1273d None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:00.819972 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24e29a8c-1cc4-4abf-b1ff-6cd2ebf1273d None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:00.820243 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24e29a8c-1cc4-4abf-b1ff-6cd2ebf1273d None admin] PATH_INFO: `/v3/domains/default` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:00.821345 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24e29a8c-1cc4-4abf-b1ff-6cd2ebf1273d None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:53:00.822816 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24e29a8c-1cc4-4abf-b1ff-6cd2ebf1273d None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:53:00.824772 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 41/80] 10.222.0.50 () {56 vars in 1108 bytes} [Tue Sep 13 15:53:00 2022] GET /identity/v3/domains/default => generated 199 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:53:00.825541 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24e29a8c-1cc4-4abf-b1ff-6cd2ebf1273d None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:53:00.835026 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cda10665-b993-4ef3-bb4f-b0f3c036ca5e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:53:00.835649 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cda10665-b993-4ef3-bb4f-b0f3c036ca5e None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:00.835936 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cda10665-b993-4ef3-bb4f-b0f3c036ca5e None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:00.836218 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cda10665-b993-4ef3-bb4f-b0f3c036ca5e None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:00.902694 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cda10665-b993-4ef3-bb4f-b0f3c036ca5e None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:53:00.904164 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cda10665-b993-4ef3-bb4f-b0f3c036ca5e None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:53:00.922506 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:53:00.923901 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24e29a8c-1cc4-4abf-b1ff-6cd2ebf1273d None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:53:00.957888 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df081f8d-2662-4903-bf1a-539f2bf4550b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:53:00.958493 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df081f8d-2662-4903-bf1a-539f2bf4550b None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:00.958779 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df081f8d-2662-4903-bf1a-539f2bf4550b None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:00.959100 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df081f8d-2662-4903-bf1a-539f2bf4550b None admin] PATH_INFO: `/v3/roles` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:00.959770 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df081f8d-2662-4903-bf1a-539f2bf4550b None admin] RBAC: Adding query filter params (name=member) {{(pid=81437) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:53:00.960077 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df081f8d-2662-4903-bf1a-539f2bf4550b None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:53:00.961484 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df081f8d-2662-4903-bf1a-539f2bf4550b None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:53:00.970094 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 42/81] 10.222.0.50 () {56 vars in 1101 bytes} [Tue Sep 13 15:53:00 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 145 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:53:00.970892 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df081f8d-2662-4903-bf1a-539f2bf4550b None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:53:00.995804 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:53:01.008762 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 40/82] 10.222.0.50 () {58 vars in 1282 bytes} [Tue Sep 13 15:53:00 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 313 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 13 15:53:01.009916 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cda10665-b993-4ef3-bb4f-b0f3c036ca5e None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:53:01.076783 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df081f8d-2662-4903-bf1a-539f2bf4550b None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:53:01.111482 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59f21b11-2778-455f-8da7-aef5a6d90d54 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:53:01.112110 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59f21b11-2778-455f-8da7-aef5a6d90d54 None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:01.112389 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59f21b11-2778-455f-8da7-aef5a6d90d54 None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:01.112700 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59f21b11-2778-455f-8da7-aef5a6d90d54 None admin] PATH_INFO: `/v3/users/5c4340db782249918182a2e21023cc55` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:01.113802 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59f21b11-2778-455f-8da7-aef5a6d90d54 None admin] RBAC: Authorizing `identity:get_user(user_id=5c4340db782249918182a2e21023cc55)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:53:01.115284 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59f21b11-2778-455f-8da7-aef5a6d90d54 None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:53:01.116842 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 43/83] 10.222.0.50 () {56 vars in 1177 bytes} [Tue Sep 13 15:53:00 2022] GET /identity/v3/users/5c4340db782249918182a2e21023cc55 => generated 275 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:53:01.121291 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cda10665-b993-4ef3-bb4f-b0f3c036ca5e None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:53:01.123411 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59f21b11-2778-455f-8da7-aef5a6d90d54 None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:53:01.139021 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:53:01.158669 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9ca5b99-e429-4408-8310-03630828e0b5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:53:01.159353 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9ca5b99-e429-4408-8310-03630828e0b5 None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:01.159638 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9ca5b99-e429-4408-8310-03630828e0b5 None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:01.159915 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9ca5b99-e429-4408-8310-03630828e0b5 None admin] PATH_INFO: `/v3/users/e0a915a0eb3c42c7af97dcc7eed5b42f` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:01.161041 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9ca5b99-e429-4408-8310-03630828e0b5 None admin] RBAC: Authorizing `identity:get_user(user_id=e0a915a0eb3c42c7af97dcc7eed5b42f)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:53:01.162921 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9ca5b99-e429-4408-8310-03630828e0b5 None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:53:01.164901 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 41/84] 10.222.0.50 () {56 vars in 1177 bytes} [Tue Sep 13 15:53:01 2022] GET /identity/v3/users/e0a915a0eb3c42c7af97dcc7eed5b42f => generated 247 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:53:01.166078 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9ca5b99-e429-4408-8310-03630828e0b5 None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:53:01.215383 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:53:01.224711 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59f21b11-2778-455f-8da7-aef5a6d90d54 None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:53:01.258084 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea4ac705-6cbc-4ae5-8d8d-3d5c2df616a2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:53:01.258694 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea4ac705-6cbc-4ae5-8d8d-3d5c2df616a2 None admin] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:01.259009 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea4ac705-6cbc-4ae5-8d8d-3d5c2df616a2 None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:01.259293 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea4ac705-6cbc-4ae5-8d8d-3d5c2df616a2 None admin] PATH_INFO: `/v3/users` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:01.259938 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea4ac705-6cbc-4ae5-8d8d-3d5c2df616a2 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:53:01.262249 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea4ac705-6cbc-4ae5-8d8d-3d5c2df616a2 None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:53:01.265214 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9ca5b99-e429-4408-8310-03630828e0b5 None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:53:01.288361 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 44/85] 10.222.0.50 () {60 vars in 1132 bytes} [Tue Sep 13 15:53:01 2022] POST /identity/v3/users => generated 289 bytes in 171 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 15:53:01.289176 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea4ac705-6cbc-4ae5-8d8d-3d5c2df616a2 None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:53:01.300475 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2885e466-4fdf-49e0-9666-ff3575fdfdde None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:53:01.301139 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2885e466-4fdf-49e0-9666-ff3575fdfdde None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:01.301419 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2885e466-4fdf-49e0-9666-ff3575fdfdde None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:01.301696 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2885e466-4fdf-49e0-9666-ff3575fdfdde None admin] PATH_INFO: `/v3/users/5c4340db782249918182a2e21023cc55` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:01.303139 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2885e466-4fdf-49e0-9666-ff3575fdfdde None admin] RBAC: Authorizing `identity:get_user(user_id=5c4340db782249918182a2e21023cc55)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:53:01.304726 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2885e466-4fdf-49e0-9666-ff3575fdfdde None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:53:01.311922 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 42/86] 10.222.0.50 () {56 vars in 1177 bytes} [Tue Sep 13 15:53:01 2022] GET /identity/v3/users/5c4340db782249918182a2e21023cc55 => generated 275 bytes in 142 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:53:01.311922 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2885e466-4fdf-49e0-9666-ff3575fdfdde None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:53:01.384907 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:53:01.411636 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea4ac705-6cbc-4ae5-8d8d-3d5c2df616a2 None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:53:01.442013 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2885e466-4fdf-49e0-9666-ff3575fdfdde None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:53:01.450637 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14087c7d-25c6-424b-86fc-3f3582632c59 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:53:01.451335 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14087c7d-25c6-424b-86fc-3f3582632c59 None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:01.451616 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14087c7d-25c6-424b-86fc-3f3582632c59 None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:01.451899 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14087c7d-25c6-424b-86fc-3f3582632c59 None admin] PATH_INFO: `/v3/projects/423edf83b2b24540924e990bb324af1e` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:01.453074 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14087c7d-25c6-424b-86fc-3f3582632c59 None admin] RBAC: Authorizing `identity:get_project(project_id=423edf83b2b24540924e990bb324af1e)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:53:01.454674 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14087c7d-25c6-424b-86fc-3f3582632c59 None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:53:01.456266 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 45/87] 10.222.0.50 () {56 vars in 1186 bytes} [Tue Sep 13 15:53:01 2022] GET /identity/v3/projects/423edf83b2b24540924e990bb324af1e => generated 298 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:53:01.457042 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14087c7d-25c6-424b-86fc-3f3582632c59 None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:53:01.482098 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:53:01.493520 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e63c206-e567-4d31-9f9f-c64424344f45 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:53:01.494141 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e63c206-e567-4d31-9f9f-c64424344f45 None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:01.494426 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e63c206-e567-4d31-9f9f-c64424344f45 None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:01.494759 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e63c206-e567-4d31-9f9f-c64424344f45 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:01.579512 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e63c206-e567-4d31-9f9f-c64424344f45 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:53:01.581956 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e63c206-e567-4d31-9f9f-c64424344f45 None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:53:01.585945 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14087c7d-25c6-424b-86fc-3f3582632c59 None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:53:01.636892 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2e5b808-2d28-4682-8f61-3ca28e5803dc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:53:01.637441 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2e5b808-2d28-4682-8f61-3ca28e5803dc None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:01.638119 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2e5b808-2d28-4682-8f61-3ca28e5803dc None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:01.638119 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2e5b808-2d28-4682-8f61-3ca28e5803dc None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:01.657767 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:53:01.712847 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2e5b808-2d28-4682-8f61-3ca28e5803dc None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:53:01.715506 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2e5b808-2d28-4682-8f61-3ca28e5803dc None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:53:01.718063 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 43/88] 10.222.0.50 () {58 vars in 1282 bytes} [Tue Sep 13 15:53:01 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 410 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 13 15:53:01.719328 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e63c206-e567-4d31-9f9f-c64424344f45 None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:53:01.734135 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:53:01.823147 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 46/89] 10.222.0.50 () {58 vars in 1282 bytes} [Tue Sep 13 15:53:01 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 366 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 13 15:53:01.824326 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2e5b808-2d28-4682-8f61-3ca28e5803dc None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:53:01.839788 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e63c206-e567-4d31-9f9f-c64424344f45 None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:53:01.887826 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e882f88-3864-4e6a-bf83-8a37f73496bd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:53:01.888855 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e882f88-3864-4e6a-bf83-8a37f73496bd None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:01.889147 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e882f88-3864-4e6a-bf83-8a37f73496bd None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:01.889429 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e882f88-3864-4e6a-bf83-8a37f73496bd None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:01.919215 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:53:01.967280 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2e5b808-2d28-4682-8f61-3ca28e5803dc None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:53:01.968299 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e882f88-3864-4e6a-bf83-8a37f73496bd None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:53:01.970169 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e882f88-3864-4e6a-bf83-8a37f73496bd None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:53:01.984578 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:53:02.003097 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-939879ec-fef1-489c-b985-b7f91c9def01 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:53:02.003721 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-939879ec-fef1-489c-b985-b7f91c9def01 None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:02.004004 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-939879ec-fef1-489c-b985-b7f91c9def01 None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:02.004283 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-939879ec-fef1-489c-b985-b7f91c9def01 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:02.005016 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-939879ec-fef1-489c-b985-b7f91c9def01 None admin] RBAC: Adding query filter params (role.id=15db4aaca3264ae29bde38903ba43471, scope.project.id=423edf83b2b24540924e990bb324af1e, user.id=5c4340db782249918182a2e21023cc55) {{(pid=81437) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:53:02.005335 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-939879ec-fef1-489c-b985-b7f91c9def01 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:53:02.006794 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-939879ec-fef1-489c-b985-b7f91c9def01 None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:53:02.019112 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 47/90] 10.222.0.50 () {56 vars in 1374 bytes} [Tue Sep 13 15:53:01 2022] GET /identity/v3/role_assignments?user.id=5c4340db782249918182a2e21023cc55&scope.project.id=423edf83b2b24540924e990bb324af1e&role.id=15db4aaca3264ae29bde38903ba43471 => generated 259 bytes in 195 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:53:02.020138 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-939879ec-fef1-489c-b985-b7f91c9def01 None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:53:02.115603 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 44/91] 10.222.0.50 () {58 vars in 1282 bytes} [Tue Sep 13 15:53:01 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 397 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 13 15:53:02.116503 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e882f88-3864-4e6a-bf83-8a37f73496bd None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:53:02.154844 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-939879ec-fef1-489c-b985-b7f91c9def01 None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:53:02.196551 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c665242-77f5-49b1-8d7f-e3e51d8883bd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:53:02.197405 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c665242-77f5-49b1-8d7f-e3e51d8883bd None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:02.197669 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:53:02.197769 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c665242-77f5-49b1-8d7f-e3e51d8883bd None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:02.198086 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c665242-77f5-49b1-8d7f-e3e51d8883bd None admin] PATH_INFO: `/v3/projects/423edf83b2b24540924e990bb324af1e` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:02.199752 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c665242-77f5-49b1-8d7f-e3e51d8883bd None admin] RBAC: Authorizing `identity:get_project(project_id=423edf83b2b24540924e990bb324af1e)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:53:02.201256 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c665242-77f5-49b1-8d7f-e3e51d8883bd None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:53:02.203018 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 48/92] 10.222.0.50 () {56 vars in 1186 bytes} [Tue Sep 13 15:53:02 2022] GET /identity/v3/projects/423edf83b2b24540924e990bb324af1e => generated 298 bytes in 183 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:53:02.203828 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c665242-77f5-49b1-8d7f-e3e51d8883bd None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:53:02.279319 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:53:02.297759 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e882f88-3864-4e6a-bf83-8a37f73496bd None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:53:02.356741 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c665242-77f5-49b1-8d7f-e3e51d8883bd None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:53:02.367569 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03252ebe-61f9-4394-b36a-821a475a6c71 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:53:02.368243 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03252ebe-61f9-4394-b36a-821a475a6c71 None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:02.368528 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03252ebe-61f9-4394-b36a-821a475a6c71 None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:02.368807 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03252ebe-61f9-4394-b36a-821a475a6c71 None admin] PATH_INFO: `/v3/projects/423edf83b2b24540924e990bb324af1e` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:02.370709 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03252ebe-61f9-4394-b36a-821a475a6c71 None admin] RBAC: Authorizing `identity:get_project(project_id=423edf83b2b24540924e990bb324af1e)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:53:02.372267 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03252ebe-61f9-4394-b36a-821a475a6c71 None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:53:02.374026 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 45/93] 10.222.0.50 () {56 vars in 1186 bytes} [Tue Sep 13 15:53:02 2022] GET /identity/v3/projects/423edf83b2b24540924e990bb324af1e => generated 298 bytes in 258 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:53:02.374820 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03252ebe-61f9-4394-b36a-821a475a6c71 None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:53:02.422582 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19bac8b8-dbcd-47d7-b9eb-859d3c240536 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:53:02.423265 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19bac8b8-dbcd-47d7-b9eb-859d3c240536 None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:02.423455 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19bac8b8-dbcd-47d7-b9eb-859d3c240536 None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:02.423734 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19bac8b8-dbcd-47d7-b9eb-859d3c240536 None admin] PATH_INFO: `/v3/projects/1df61ac8263c4c1e9847d7bdfbcdc627` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:02.425486 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19bac8b8-dbcd-47d7-b9eb-859d3c240536 None admin] RBAC: Authorizing `identity:get_project(project_id=1df61ac8263c4c1e9847d7bdfbcdc627)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:53:02.426960 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19bac8b8-dbcd-47d7-b9eb-859d3c240536 None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:53:02.428607 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 49/94] 10.222.0.50 () {56 vars in 1186 bytes} [Tue Sep 13 15:53:02 2022] GET /identity/v3/projects/1df61ac8263c4c1e9847d7bdfbcdc627 => generated 312 bytes in 225 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:53:02.429373 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19bac8b8-dbcd-47d7-b9eb-859d3c240536 None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:53:02.572184 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19bac8b8-dbcd-47d7-b9eb-859d3c240536 None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:53:02.595635 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03252ebe-61f9-4394-b36a-821a475a6c71 None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:53:02.604775 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:53:02.626293 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:53:02.639406 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cca9d84-0ea2-40da-8778-ceca3bd7e051 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:53:02.640031 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cca9d84-0ea2-40da-8778-ceca3bd7e051 None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:02.640316 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cca9d84-0ea2-40da-8778-ceca3bd7e051 None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:02.640589 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cca9d84-0ea2-40da-8778-ceca3bd7e051 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:02.641324 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8cca9d84-0ea2-40da-8778-ceca3bd7e051 None admin] RBAC: Adding query filter params (role.id=031d4e1a23e34f71858fc4a6dfbd4ccb, scope.project.id=423edf83b2b24540924e990bb324af1e, user.id=e0a915a0eb3c42c7af97dcc7eed5b42f) {{(pid=81437) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:53:02.641634 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8cca9d84-0ea2-40da-8778-ceca3bd7e051 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:53:02.643106 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8cca9d84-0ea2-40da-8778-ceca3bd7e051 None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:53:02.649508 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49933c08-b5aa-4a9d-948d-a43e99e305cd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:53:02.650131 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49933c08-b5aa-4a9d-948d-a43e99e305cd None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:02.650413 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49933c08-b5aa-4a9d-948d-a43e99e305cd None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:02.650687 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49933c08-b5aa-4a9d-948d-a43e99e305cd None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:02.651529 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49933c08-b5aa-4a9d-948d-a43e99e305cd None admin] RBAC: Adding query filter params (role.id=f74e3f6c20bb426ca030f211c3a9aef5, scope.project.id=423edf83b2b24540924e990bb324af1e, user.id=5c4340db782249918182a2e21023cc55) {{(pid=81438) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:53:02.651845 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49933c08-b5aa-4a9d-948d-a43e99e305cd None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:53:02.653282 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49933c08-b5aa-4a9d-948d-a43e99e305cd None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:53:02.658253 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 50/95] 10.222.0.50 () {56 vars in 1374 bytes} [Tue Sep 13 15:53:02 2022] GET /identity/v3/role_assignments?user.id=e0a915a0eb3c42c7af97dcc7eed5b42f&scope.project.id=423edf83b2b24540924e990bb324af1e&role.id=031d4e1a23e34f71858fc4a6dfbd4ccb => generated 259 bytes in 229 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:53:02.659118 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cca9d84-0ea2-40da-8778-ceca3bd7e051 None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:53:02.674721 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 46/96] 10.222.0.50 () {56 vars in 1374 bytes} [Tue Sep 13 15:53:02 2022] GET /identity/v3/role_assignments?user.id=5c4340db782249918182a2e21023cc55&scope.project.id=423edf83b2b24540924e990bb324af1e&role.id=f74e3f6c20bb426ca030f211c3a9aef5 => generated 259 bytes in 297 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:53:02.828979 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cca9d84-0ea2-40da-8778-ceca3bd7e051 None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:53:02.880415 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78826747-a8ae-4a86-8a23-9de3aefe69de None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:53:02.881090 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78826747-a8ae-4a86-8a23-9de3aefe69de None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:02.881385 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78826747-a8ae-4a86-8a23-9de3aefe69de None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:02.881661 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78826747-a8ae-4a86-8a23-9de3aefe69de None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:02.882459 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78826747-a8ae-4a86-8a23-9de3aefe69de None admin] RBAC: Adding query filter params (role.id=15db4aaca3264ae29bde38903ba43471, scope.project.id=1df61ac8263c4c1e9847d7bdfbcdc627, user.id=5c4340db782249918182a2e21023cc55) {{(pid=81437) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:53:02.882776 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78826747-a8ae-4a86-8a23-9de3aefe69de None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:53:02.884333 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78826747-a8ae-4a86-8a23-9de3aefe69de None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:53:02.891567 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 51/97] 10.222.0.50 () {56 vars in 1374 bytes} [Tue Sep 13 15:53:02 2022] GET /identity/v3/role_assignments?user.id=5c4340db782249918182a2e21023cc55&scope.project.id=1df61ac8263c4c1e9847d7bdfbcdc627&role.id=15db4aaca3264ae29bde38903ba43471 => generated 259 bytes in 233 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:53:04.238867 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78382c26-6c7b-49a4-8c04-058fab99b711 None None] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:04.239395 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78382c26-6c7b-49a4-8c04-058fab99b711 None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:04.239499 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78382c26-6c7b-49a4-8c04-058fab99b711 None None] PATH_INFO: `/` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:04.240951 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 47/98] 10.222.0.50 () {54 vars in 927 bytes} [Tue Sep 13 15:53:04 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:53:04.250738 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27272f47-ed1a-43e4-b8b9-f388941073df None None] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:04.251181 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27272f47-ed1a-43e4-b8b9-f388941073df None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:04.251367 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27272f47-ed1a-43e4-b8b9-f388941073df None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:04.276783 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.auth.core [None req-27272f47-ed1a-43e4-b8b9-f388941073df None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81437) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:53:04.277847 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-27272f47-ed1a-43e4-b8b9-f388941073df None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:53:04.314144 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e54a4b04-5b12-4c72-8db8-8d990e2e2430 None None] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:04.314415 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e54a4b04-5b12-4c72-8db8-8d990e2e2430 None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:04.314635 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e54a4b04-5b12-4c72-8db8-8d990e2e2430 None None] PATH_INFO: `/` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:04.316016 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 48/99] 10.222.0.50 () {54 vars in 927 bytes} [Tue Sep 13 15:53:04 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:53:04.321959 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0360c309-1b12-4212-96cb-82ac88387741 None None] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:04.322246 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0360c309-1b12-4212-96cb-82ac88387741 None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:04.322529 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0360c309-1b12-4212-96cb-82ac88387741 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:04.335697 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:53:04.353859 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.auth.core [None req-0360c309-1b12-4212-96cb-82ac88387741 None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81438) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:53:04.355698 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-0360c309-1b12-4212-96cb-82ac88387741 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:53:04.387347 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 52/100] 10.222.0.50 () {58 vars in 1024 bytes} [Tue Sep 13 15:53:04 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 137 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 13 15:53:04.441437 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 49/101] 10.222.0.50 () {58 vars in 1024 bytes} [Tue Sep 13 15:53:04 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 120 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 13 15:53:04.859245 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27272f47-ed1a-43e4-b8b9-f388941073df None None] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:53:04.926772 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0360c309-1b12-4212-96cb-82ac88387741 None None] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:53:04.966919 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27272f47-ed1a-43e4-b8b9-f388941073df None None] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:53:05.007752 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-344436ac-07ae-489e-9154-ae6f810f7f80 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:53:05.008409 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-344436ac-07ae-489e-9154-ae6f810f7f80 None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:05.008688 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-344436ac-07ae-489e-9154-ae6f810f7f80 None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:05.008960 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-344436ac-07ae-489e-9154-ae6f810f7f80 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:05.027770 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-344436ac-07ae-489e-9154-ae6f810f7f80 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 13 15:53:05.037220 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application [None req-344436ac-07ae-489e-9154-ae6f810f7f80 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Sep 13 15:53:05.037220 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:53:05.037220 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:53:05.037220 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:53:05.037220 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:53:05.037220 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:53:05.037220 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:53:05.037220 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:53:05.037220 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:53:05.037220 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:53:05.037220 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:53:05.037220 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:53:05.037220 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 13 15:53:05.037220 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 13 15:53:05.037220 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 13 15:53:05.037220 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 13 15:53:05.038077 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 13 15:53:05.038077 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 13 15:53:05.038077 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 13 15:53:05.038077 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 13 15:53:05.038077 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:53:05.038077 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:53:05.038077 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:53:05.038077 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:53:05.038077 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:53:05.038077 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:53:05.038077 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:53:05.038077 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application with Lock( Sep 13 15:53:05.038077 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:53:05.038077 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:53:05.038077 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:53:05.038077 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:53:05.038819 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:53:05.038819 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:53:05.038819 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:53:05.038819 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:53:05.038819 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 13 15:53:05.038819 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 13 15:53:05.038819 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 13 15:53:05.038819 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 13 15:53:05.038819 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 13 15:53:05.038819 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 13 15:53:05.038819 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Sep 13 15:53:05.038819 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application Sep 13 15:53:05.039900 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 53/102] 10.222.0.50 () {56 vars in 1099 bytes} [Tue Sep 13 15:53:04 2022] GET /identity/v3/roles/reader => generated 84 bytes in 180 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:53:05.045260 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-344436ac-07ae-489e-9154-ae6f810f7f80 None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:53:05.111367 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:53:05.209806 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0360c309-1b12-4212-96cb-82ac88387741 None None] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:53:05.234815 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-344436ac-07ae-489e-9154-ae6f810f7f80 None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:53:05.236707 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:53:05.270142 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4e216d8-aac0-4505-bd91-3b13645cbf82 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:53:05.270791 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4e216d8-aac0-4505-bd91-3b13645cbf82 None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:05.271137 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4e216d8-aac0-4505-bd91-3b13645cbf82 None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:05.271422 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4e216d8-aac0-4505-bd91-3b13645cbf82 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:05.272996 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4e216d8-aac0-4505-bd91-3b13645cbf82 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:53:05.274443 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4e216d8-aac0-4505-bd91-3b13645cbf82 None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:53:05.276086 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 50/103] 10.222.0.50 () {56 vars in 1108 bytes} [Tue Sep 13 15:53:04 2022] GET /identity/v3/domains/default => generated 199 bytes in 350 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 13 15:53:05.279890 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b59dd215-5d20-4b26-89ca-aa5a51c4aa25 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:53:05.280509 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b59dd215-5d20-4b26-89ca-aa5a51c4aa25 None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:05.281058 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b59dd215-5d20-4b26-89ca-aa5a51c4aa25 None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:05.281352 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b59dd215-5d20-4b26-89ca-aa5a51c4aa25 None admin] PATH_INFO: `/v3/roles` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:05.282033 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b59dd215-5d20-4b26-89ca-aa5a51c4aa25 None admin] RBAC: Adding query filter params (name=reader) {{(pid=81437) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:53:05.282343 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b59dd215-5d20-4b26-89ca-aa5a51c4aa25 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:53:05.283774 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b59dd215-5d20-4b26-89ca-aa5a51c4aa25 None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:53:05.285921 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4e216d8-aac0-4505-bd91-3b13645cbf82 None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:53:05.300486 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 54/104] 10.222.0.50 () {56 vars in 1101 bytes} [Tue Sep 13 15:53:05 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 256 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 13 15:53:05.308842 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b59dd215-5d20-4b26-89ca-aa5a51c4aa25 None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:53:05.499845 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4e216d8-aac0-4505-bd91-3b13645cbf82 None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:53:05.534129 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:53:05.546336 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b59dd215-5d20-4b26-89ca-aa5a51c4aa25 None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:53:05.577963 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19324685-5cf3-4289-97da-f3ac50d8e858 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:53:05.578972 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19324685-5cf3-4289-97da-f3ac50d8e858 None admin] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:05.579462 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19324685-5cf3-4289-97da-f3ac50d8e858 None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:05.579930 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19324685-5cf3-4289-97da-f3ac50d8e858 None admin] PATH_INFO: `/v3/projects` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:05.580777 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19324685-5cf3-4289-97da-f3ac50d8e858 None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:53:05.582425 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19324685-5cf3-4289-97da-f3ac50d8e858 None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:53:05.630056 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 51/105] 10.222.0.50 () {60 vars in 1141 bytes} [Tue Sep 13 15:53:05 2022] POST /identity/v3/projects => generated 302 bytes in 344 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 15:53:05.640055 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0215aa84-ae80-43e8-9c90-3e686b5fafac None None] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:05.640351 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0215aa84-ae80-43e8-9c90-3e686b5fafac None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:05.640633 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0215aa84-ae80-43e8-9c90-3e686b5fafac None None] PATH_INFO: `/` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:05.642040 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 52/106] 10.222.0.50 () {54 vars in 927 bytes} [Tue Sep 13 15:53:05 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 13 15:53:05.647916 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-539387b6-c270-4a5f-a216-4adb80f907a2 None None] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:05.648284 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-539387b6-c270-4a5f-a216-4adb80f907a2 None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:05.648642 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-539387b6-c270-4a5f-a216-4adb80f907a2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:05.668232 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ecb23cd-f0f4-44cc-a31d-a567a822b15d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:53:05.668961 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ecb23cd-f0f4-44cc-a31d-a567a822b15d None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:05.669249 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ecb23cd-f0f4-44cc-a31d-a567a822b15d None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:05.669521 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ecb23cd-f0f4-44cc-a31d-a567a822b15d None admin] PATH_INFO: `/v3/users/83ce764787ea4cc389a55bbb8d35c45f` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:05.669919 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.auth.core [None req-539387b6-c270-4a5f-a216-4adb80f907a2 None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81438) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:53:05.671303 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-539387b6-c270-4a5f-a216-4adb80f907a2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:53:05.708357 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:53:05.715691 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ecb23cd-f0f4-44cc-a31d-a567a822b15d None admin] RBAC: Authorizing `identity:get_user(user_id=83ce764787ea4cc389a55bbb8d35c45f)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:53:05.717238 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ecb23cd-f0f4-44cc-a31d-a567a822b15d None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:53:05.719042 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 55/107] 10.222.0.50 () {56 vars in 1177 bytes} [Tue Sep 13 15:53:05 2022] GET /identity/v3/users/83ce764787ea4cc389a55bbb8d35c45f => generated 289 bytes in 411 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:53:05.720337 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f143ef9-5f9d-4327-96b0-9bb3377547d3 None None] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:05.720619 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f143ef9-5f9d-4327-96b0-9bb3377547d3 None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:05.720907 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f143ef9-5f9d-4327-96b0-9bb3377547d3 None None] PATH_INFO: `/` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:05.722247 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 56/108] 10.222.0.50 () {54 vars in 927 bytes} [Tue Sep 13 15:53:05 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:53:05.731714 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f143ef9-5f9d-4327-96b0-9bb3377547d3 None None] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:53:05.774524 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 53/109] 10.222.0.50 () {58 vars in 1024 bytes} [Tue Sep 13 15:53:05 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 127 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 13 15:53:05.775985 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8abe0e6d-08d3-46aa-93b0-9e3f6552a851 None None] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:05.777697 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8abe0e6d-08d3-46aa-93b0-9e3f6552a851 None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:05.777985 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8abe0e6d-08d3-46aa-93b0-9e3f6552a851 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:05.812101 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.auth.core [None req-8abe0e6d-08d3-46aa-93b0-9e3f6552a851 None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81438) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:53:05.814765 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-8abe0e6d-08d3-46aa-93b0-9e3f6552a851 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:53:05.903347 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 54/110] 10.222.0.50 () {58 vars in 1024 bytes} [Tue Sep 13 15:53:05 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 128 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 13 15:53:05.976760 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6e8fdca-f034-4aa7-82c9-a873fa90b444 None None] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:05.977051 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6e8fdca-f034-4aa7-82c9-a873fa90b444 None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:05.977592 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6e8fdca-f034-4aa7-82c9-a873fa90b444 None None] PATH_INFO: `/` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:05.983123 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 55/111] 10.222.0.50 () {54 vars in 927 bytes} [Tue Sep 13 15:53:05 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:53:05.988375 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f143ef9-5f9d-4327-96b0-9bb3377547d3 None None] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:53:05.994235 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60b33323-2aae-4821-90e8-4a9613388c22 None None] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:05.994530 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60b33323-2aae-4821-90e8-4a9613388c22 None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:05.995771 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60b33323-2aae-4821-90e8-4a9613388c22 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:06.024412 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.auth.core [None req-60b33323-2aae-4821-90e8-4a9613388c22 None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81438) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:53:06.025710 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-60b33323-2aae-4821-90e8-4a9613388c22 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:53:06.054283 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a6dadb4-ba6a-4f42-9f8d-d47565d9220e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:53:06.054925 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a6dadb4-ba6a-4f42-9f8d-d47565d9220e None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:06.055206 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a6dadb4-ba6a-4f42-9f8d-d47565d9220e None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:06.055484 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a6dadb4-ba6a-4f42-9f8d-d47565d9220e None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:06.101945 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:53:06.153381 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 56/112] 10.222.0.50 () {58 vars in 1024 bytes} [Tue Sep 13 15:53:05 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 161 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 13 15:53:06.154274 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60b33323-2aae-4821-90e8-4a9613388c22 None None] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:53:06.170572 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a6dadb4-ba6a-4f42-9f8d-d47565d9220e None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:53:06.172140 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a6dadb4-ba6a-4f42-9f8d-d47565d9220e None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:53:06.183105 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:53:06.267047 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60b33323-2aae-4821-90e8-4a9613388c22 None None] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:53:06.284352 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 57/113] 10.222.0.50 () {58 vars in 1282 bytes} [Tue Sep 13 15:53:05 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 553 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Sep 13 15:53:06.286279 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8d2f93a-d726-4d10-86f7-4000ae91a6c0 None None] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:06.286694 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8d2f93a-d726-4d10-86f7-4000ae91a6c0 None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:06.287123 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8d2f93a-d726-4d10-86f7-4000ae91a6c0 None None] PATH_INFO: `/` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:06.294027 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 58/114] 10.222.0.50 () {54 vars in 927 bytes} [Tue Sep 13 15:53:06 2022] GET /identity => generated 270 bytes in 9 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:53:06.295681 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8d2f93a-d726-4d10-86f7-4000ae91a6c0 None None] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:53:06.306098 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffb8dbd9-f69c-4024-b139-12ae1e512851 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:53:06.307038 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffb8dbd9-f69c-4024-b139-12ae1e512851 None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:06.307902 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffb8dbd9-f69c-4024-b139-12ae1e512851 None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:06.308358 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffb8dbd9-f69c-4024-b139-12ae1e512851 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:06.319113 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffb8dbd9-f69c-4024-b139-12ae1e512851 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 13 15:53:06.324522 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application [None req-ffb8dbd9-f69c-4024-b139-12ae1e512851 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 13 15:53:06.324522 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:53:06.324522 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:53:06.324522 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:53:06.324522 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:53:06.324522 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:53:06.324522 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:53:06.324522 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:53:06.324522 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:53:06.324522 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:53:06.324522 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:53:06.324522 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:53:06.324522 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 13 15:53:06.324522 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 13 15:53:06.324522 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 13 15:53:06.324522 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 13 15:53:06.325378 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 13 15:53:06.325378 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 13 15:53:06.325378 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 13 15:53:06.325378 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 13 15:53:06.325378 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:53:06.325378 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:53:06.325378 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:53:06.325378 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:53:06.325378 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:53:06.325378 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:53:06.325378 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:53:06.325378 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application with Lock( Sep 13 15:53:06.325378 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:53:06.325378 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:53:06.325378 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:53:06.325378 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:53:06.326523 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:53:06.326523 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:53:06.326523 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:53:06.326523 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:53:06.326523 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 13 15:53:06.326523 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 13 15:53:06.326523 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 13 15:53:06.326523 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 13 15:53:06.326523 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 13 15:53:06.326523 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 13 15:53:06.326523 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 13 15:53:06.326523 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Sep 13 15:53:06.326523 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 57/115] 10.222.0.50 () {56 vars in 1099 bytes} [Tue Sep 13 15:53:06 2022] GET /identity/v3/roles/member => generated 84 bytes in 172 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:53:06.327183 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3f8f71a-903e-4bf3-802b-10cd9f95cb87 None None] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:06.331697 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3f8f71a-903e-4bf3-802b-10cd9f95cb87 None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:06.332006 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3f8f71a-903e-4bf3-802b-10cd9f95cb87 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:06.350090 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.auth.core [None req-a3f8f71a-903e-4bf3-802b-10cd9f95cb87 None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81438) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:53:06.351326 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-a3f8f71a-903e-4bf3-802b-10cd9f95cb87 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:53:06.363575 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:53:06.416462 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8d2f93a-d726-4d10-86f7-4000ae91a6c0 None None] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:53:06.421276 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 58/116] 10.222.0.50 () {58 vars in 1024 bytes} [Tue Sep 13 15:53:06 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 13 15:53:06.422143 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3f8f71a-903e-4bf3-802b-10cd9f95cb87 None None] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:53:06.442440 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:53:06.508868 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-048cc11a-fcdd-463d-8a93-2def43ddadca None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:53:06.511131 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-048cc11a-fcdd-463d-8a93-2def43ddadca None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:06.511760 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-048cc11a-fcdd-463d-8a93-2def43ddadca None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:06.512455 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-048cc11a-fcdd-463d-8a93-2def43ddadca None admin] PATH_INFO: `/v3/projects/423edf83b2b24540924e990bb324af1e` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:06.514030 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-048cc11a-fcdd-463d-8a93-2def43ddadca None admin] RBAC: Authorizing `identity:get_project(project_id=423edf83b2b24540924e990bb324af1e)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:53:06.516167 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-048cc11a-fcdd-463d-8a93-2def43ddadca None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:53:06.518555 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 59/117] 10.222.0.50 () {56 vars in 1186 bytes} [Tue Sep 13 15:53:06 2022] GET /identity/v3/projects/423edf83b2b24540924e990bb324af1e => generated 298 bytes in 223 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 13 15:53:06.523795 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-048cc11a-fcdd-463d-8a93-2def43ddadca None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:53:06.537833 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3f8f71a-903e-4bf3-802b-10cd9f95cb87 None None] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:53:06.579014 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd84b3ed-17c8-4a40-99e9-830fa2371621 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:53:06.580015 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd84b3ed-17c8-4a40-99e9-830fa2371621 None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:06.580815 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd84b3ed-17c8-4a40-99e9-830fa2371621 None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:06.581099 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd84b3ed-17c8-4a40-99e9-830fa2371621 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:06.594001 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd84b3ed-17c8-4a40-99e9-830fa2371621 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 13 15:53:06.600269 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application [None req-dd84b3ed-17c8-4a40-99e9-830fa2371621 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 13 15:53:06.600269 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:53:06.600269 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:53:06.600269 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:53:06.600269 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:53:06.600269 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:53:06.600269 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:53:06.600269 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:53:06.600269 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:53:06.600269 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:53:06.600269 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:53:06.600269 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:53:06.600269 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 13 15:53:06.600269 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 13 15:53:06.600269 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 13 15:53:06.600269 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 13 15:53:06.601114 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 13 15:53:06.601114 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 13 15:53:06.601114 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 13 15:53:06.601114 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 13 15:53:06.601114 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:53:06.601114 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:53:06.601114 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:53:06.601114 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:53:06.601114 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:53:06.601114 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:53:06.601114 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:53:06.601114 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application with Lock( Sep 13 15:53:06.601114 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:53:06.601114 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:53:06.601114 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:53:06.601114 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:53:06.601870 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:53:06.601870 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:53:06.601870 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:53:06.601870 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:53:06.601870 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 13 15:53:06.601870 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 13 15:53:06.601870 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 13 15:53:06.601870 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 13 15:53:06.601870 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 13 15:53:06.601870 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 13 15:53:06.601870 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 13 15:53:06.601870 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Sep 13 15:53:06.602464 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 59/118] 10.222.0.50 () {56 vars in 1099 bytes} [Tue Sep 13 15:53:06 2022] GET /identity/v3/roles/member => generated 84 bytes in 180 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:53:06.606921 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd84b3ed-17c8-4a40-99e9-830fa2371621 None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:53:06.645469 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-048cc11a-fcdd-463d-8a93-2def43ddadca None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:53:06.669746 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:53:06.682391 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbba1a06-eb20-42d3-947e-057d5389486d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:53:06.683050 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbba1a06-eb20-42d3-947e-057d5389486d None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:06.683328 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbba1a06-eb20-42d3-947e-057d5389486d None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:06.683603 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbba1a06-eb20-42d3-947e-057d5389486d None admin] PATH_INFO: `/v3/roles` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:06.684266 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbba1a06-eb20-42d3-947e-057d5389486d None admin] RBAC: Adding query filter params (name=member) {{(pid=81437) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:53:06.684570 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbba1a06-eb20-42d3-947e-057d5389486d None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:53:06.685975 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbba1a06-eb20-42d3-947e-057d5389486d None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:53:06.695811 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 60/119] 10.222.0.50 () {56 vars in 1101 bytes} [Tue Sep 13 15:53:06 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:53:06.701125 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbba1a06-eb20-42d3-947e-057d5389486d None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:53:06.706076 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:53:06.736166 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd84b3ed-17c8-4a40-99e9-830fa2371621 None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:53:06.773979 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b03dc211-462a-4a88-b1cf-6ff1aa09bb8e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:53:06.774627 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b03dc211-462a-4a88-b1cf-6ff1aa09bb8e None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:06.774930 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b03dc211-462a-4a88-b1cf-6ff1aa09bb8e None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:06.775223 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b03dc211-462a-4a88-b1cf-6ff1aa09bb8e None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:06.786407 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b03dc211-462a-4a88-b1cf-6ff1aa09bb8e None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 13 15:53:06.791868 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application [None req-b03dc211-462a-4a88-b1cf-6ff1aa09bb8e None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 13 15:53:06.791868 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:53:06.791868 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:53:06.791868 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:53:06.791868 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:53:06.791868 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:53:06.791868 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:53:06.791868 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:53:06.791868 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:53:06.791868 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:53:06.791868 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:53:06.791868 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:53:06.791868 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 13 15:53:06.791868 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 13 15:53:06.791868 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 13 15:53:06.791868 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 13 15:53:06.792735 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 13 15:53:06.792735 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 13 15:53:06.792735 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 13 15:53:06.792735 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 13 15:53:06.792735 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:53:06.792735 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:53:06.792735 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:53:06.792735 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:53:06.792735 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:53:06.792735 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:53:06.792735 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:53:06.792735 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application with Lock( Sep 13 15:53:06.792735 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:53:06.792735 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:53:06.792735 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:53:06.792735 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:53:06.793937 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:53:06.793937 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:53:06.793937 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:53:06.793937 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:53:06.793937 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 13 15:53:06.793937 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 13 15:53:06.793937 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 13 15:53:06.793937 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 13 15:53:06.793937 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 13 15:53:06.793937 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 13 15:53:06.793937 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 13 15:53:06.793937 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Sep 13 15:53:06.793937 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 60/120] 10.222.0.50 () {56 vars in 1114 bytes} [Tue Sep 13 15:53:06 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 191 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:53:06.794803 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b03dc211-462a-4a88-b1cf-6ff1aa09bb8e None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:53:06.810387 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbba1a06-eb20-42d3-947e-057d5389486d None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:53:06.848750 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3913b418-28bd-48fa-9e84-625a8ef96971 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:53:06.849358 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3913b418-28bd-48fa-9e84-625a8ef96971 None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:06.849629 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3913b418-28bd-48fa-9e84-625a8ef96971 None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:06.849900 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3913b418-28bd-48fa-9e84-625a8ef96971 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:06.850621 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3913b418-28bd-48fa-9e84-625a8ef96971 None admin] RBAC: Adding query filter params (role.id=b2d1670446a54085ad2bdb935f003d25, scope.project.id=423edf83b2b24540924e990bb324af1e, user.id=83ce764787ea4cc389a55bbb8d35c45f) {{(pid=81437) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:53:06.850946 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3913b418-28bd-48fa-9e84-625a8ef96971 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:53:06.852390 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3913b418-28bd-48fa-9e84-625a8ef96971 None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:53:06.866934 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 61/121] 10.222.0.50 () {56 vars in 1374 bytes} [Tue Sep 13 15:53:06 2022] GET /identity/v3/role_assignments?user.id=83ce764787ea4cc389a55bbb8d35c45f&scope.project.id=423edf83b2b24540924e990bb324af1e&role.id=b2d1670446a54085ad2bdb935f003d25 => generated 259 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:53:06.866934 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3913b418-28bd-48fa-9e84-625a8ef96971 None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:53:06.918254 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b03dc211-462a-4a88-b1cf-6ff1aa09bb8e None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:53:06.929969 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:53:06.960707 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89f1ab7c-1b1f-4859-bb70-b85793c0bcac None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:53:06.961542 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89f1ab7c-1b1f-4859-bb70-b85793c0bcac None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:06.961822 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89f1ab7c-1b1f-4859-bb70-b85793c0bcac None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:06.962095 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89f1ab7c-1b1f-4859-bb70-b85793c0bcac None admin] PATH_INFO: `/v3/roles` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:06.962767 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89f1ab7c-1b1f-4859-bb70-b85793c0bcac None admin] RBAC: Adding query filter params (name=member) {{(pid=81438) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:53:06.963100 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89f1ab7c-1b1f-4859-bb70-b85793c0bcac None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:53:06.965313 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89f1ab7c-1b1f-4859-bb70-b85793c0bcac None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:53:06.968420 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:53:06.983126 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 61/122] 10.222.0.50 () {56 vars in 1101 bytes} [Tue Sep 13 15:53:06 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:53:06.983973 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89f1ab7c-1b1f-4859-bb70-b85793c0bcac None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:53:07.013940 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3913b418-28bd-48fa-9e84-625a8ef96971 None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:53:07.064815 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85a2d63e-a88e-45b2-9fb7-0014fcdb4635 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:53:07.066474 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85a2d63e-a88e-45b2-9fb7-0014fcdb4635 None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:07.067279 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85a2d63e-a88e-45b2-9fb7-0014fcdb4635 None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:07.067585 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85a2d63e-a88e-45b2-9fb7-0014fcdb4635 None admin] PATH_INFO: `/v3/users/5c4340db782249918182a2e21023cc55` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:07.068808 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85a2d63e-a88e-45b2-9fb7-0014fcdb4635 None admin] RBAC: Authorizing `identity:get_user(user_id=5c4340db782249918182a2e21023cc55)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:53:07.070977 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85a2d63e-a88e-45b2-9fb7-0014fcdb4635 None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:53:07.074349 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 62/123] 10.222.0.50 () {56 vars in 1177 bytes} [Tue Sep 13 15:53:06 2022] GET /identity/v3/users/5c4340db782249918182a2e21023cc55 => generated 275 bytes in 216 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:53:07.075655 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85a2d63e-a88e-45b2-9fb7-0014fcdb4635 None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:53:07.113404 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89f1ab7c-1b1f-4859-bb70-b85793c0bcac None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:53:07.151396 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec1e61dc-e299-4a99-9126-48a1121f0fa3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:53:07.152041 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec1e61dc-e299-4a99-9126-48a1121f0fa3 None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:07.152322 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec1e61dc-e299-4a99-9126-48a1121f0fa3 None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:07.152600 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec1e61dc-e299-4a99-9126-48a1121f0fa3 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:07.166001 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec1e61dc-e299-4a99-9126-48a1121f0fa3 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 13 15:53:07.172257 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application [None req-ec1e61dc-e299-4a99-9126-48a1121f0fa3 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 13 15:53:07.172257 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:53:07.172257 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:53:07.172257 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:53:07.172257 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:53:07.172257 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:53:07.172257 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:53:07.172257 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:53:07.172257 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:53:07.172257 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:53:07.172257 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:53:07.172257 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:53:07.172257 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 13 15:53:07.172257 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 13 15:53:07.172257 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 13 15:53:07.172257 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 13 15:53:07.173115 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 13 15:53:07.173115 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 13 15:53:07.173115 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 13 15:53:07.173115 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 13 15:53:07.173115 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:53:07.173115 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:53:07.173115 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:53:07.173115 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:53:07.173115 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:53:07.173115 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:53:07.173115 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:53:07.173115 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application with Lock( Sep 13 15:53:07.173115 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:53:07.173115 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:53:07.173115 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:53:07.173115 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:53:07.173913 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:53:07.173913 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:53:07.173913 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:53:07.173913 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:53:07.173913 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 13 15:53:07.173913 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 13 15:53:07.173913 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 13 15:53:07.173913 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 13 15:53:07.173913 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 13 15:53:07.173913 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 13 15:53:07.173913 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 13 15:53:07.173913 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Sep 13 15:53:07.174502 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 62/124] 10.222.0.50 () {56 vars in 1096 bytes} [Tue Sep 13 15:53:06 2022] GET /identity/v3/roles/admin => generated 83 bytes in 191 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:53:07.180373 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec1e61dc-e299-4a99-9126-48a1121f0fa3 None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:53:07.186843 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85a2d63e-a88e-45b2-9fb7-0014fcdb4635 None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:53:07.206327 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:53:07.227548 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1b19296-ef74-4f53-9d1f-36cba18673aa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:53:07.228201 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1b19296-ef74-4f53-9d1f-36cba18673aa None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:07.228488 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1b19296-ef74-4f53-9d1f-36cba18673aa None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:07.228821 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1b19296-ef74-4f53-9d1f-36cba18673aa None admin] PATH_INFO: `/v3/roles` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:07.229789 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1b19296-ef74-4f53-9d1f-36cba18673aa None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81437) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:53:07.230097 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1b19296-ef74-4f53-9d1f-36cba18673aa None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:53:07.231810 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1b19296-ef74-4f53-9d1f-36cba18673aa None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:53:07.241177 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 63/125] 10.222.0.50 () {56 vars in 1111 bytes} [Tue Sep 13 15:53:07 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:53:07.242012 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1b19296-ef74-4f53-9d1f-36cba18673aa None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:53:07.252970 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:53:07.303962 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec1e61dc-e299-4a99-9126-48a1121f0fa3 None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:53:07.344937 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0868a392-a244-43a3-bd6d-223f7c5c395c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:53:07.346386 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0868a392-a244-43a3-bd6d-223f7c5c395c None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:07.347063 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0868a392-a244-43a3-bd6d-223f7c5c395c None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:07.347882 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0868a392-a244-43a3-bd6d-223f7c5c395c None admin] PATH_INFO: `/v3/users/5c4340db782249918182a2e21023cc55` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:07.349531 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0868a392-a244-43a3-bd6d-223f7c5c395c None admin] RBAC: Authorizing `identity:get_user(user_id=5c4340db782249918182a2e21023cc55)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:53:07.351219 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0868a392-a244-43a3-bd6d-223f7c5c395c None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:53:07.353513 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 63/126] 10.222.0.50 () {56 vars in 1177 bytes} [Tue Sep 13 15:53:07 2022] GET /identity/v3/users/5c4340db782249918182a2e21023cc55 => generated 275 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:53:07.358946 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0868a392-a244-43a3-bd6d-223f7c5c395c None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:53:07.361159 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1b19296-ef74-4f53-9d1f-36cba18673aa None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:53:07.402745 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99839a4c-9f25-49ce-9a42-95c44361cded None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:53:07.403875 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99839a4c-9f25-49ce-9a42-95c44361cded None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:07.404388 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99839a4c-9f25-49ce-9a42-95c44361cded None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:07.404867 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99839a4c-9f25-49ce-9a42-95c44361cded None admin] PATH_INFO: `/v3/projects/1df61ac8263c4c1e9847d7bdfbcdc627` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:07.406295 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99839a4c-9f25-49ce-9a42-95c44361cded None admin] RBAC: Authorizing `identity:get_project(project_id=1df61ac8263c4c1e9847d7bdfbcdc627)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:53:07.408109 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99839a4c-9f25-49ce-9a42-95c44361cded None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:53:07.410148 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 64/127] 10.222.0.50 () {56 vars in 1186 bytes} [Tue Sep 13 15:53:07 2022] GET /identity/v3/projects/1df61ac8263c4c1e9847d7bdfbcdc627 => generated 312 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:53:07.419895 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99839a4c-9f25-49ce-9a42-95c44361cded None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:53:07.473945 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0868a392-a244-43a3-bd6d-223f7c5c395c None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:53:07.488784 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:53:07.502776 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:53:07.513377 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6457d4a-9111-40d3-bc7b-21cec586f642 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:53:07.513981 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6457d4a-9111-40d3-bc7b-21cec586f642 None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:07.514261 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6457d4a-9111-40d3-bc7b-21cec586f642 None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:07.514532 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6457d4a-9111-40d3-bc7b-21cec586f642 None admin] PATH_INFO: `/v3/roles` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:07.515270 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6457d4a-9111-40d3-bc7b-21cec586f642 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81438) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:53:07.515575 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6457d4a-9111-40d3-bc7b-21cec586f642 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:53:07.516986 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6457d4a-9111-40d3-bc7b-21cec586f642 None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:53:07.525484 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 64/128] 10.222.0.50 () {56 vars in 1099 bytes} [Tue Sep 13 15:53:07 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:53:07.526329 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6457d4a-9111-40d3-bc7b-21cec586f642 None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:53:07.564000 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99839a4c-9f25-49ce-9a42-95c44361cded None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:53:07.609983 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2487851-67ee-415c-8f49-890b74500762 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:53:07.610792 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2487851-67ee-415c-8f49-890b74500762 None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:07.611122 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2487851-67ee-415c-8f49-890b74500762 None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:07.611400 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2487851-67ee-415c-8f49-890b74500762 None admin] PATH_INFO: `/v3/users/5c4340db782249918182a2e21023cc55` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:07.613451 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2487851-67ee-415c-8f49-890b74500762 None admin] RBAC: Authorizing `identity:get_user(user_id=5c4340db782249918182a2e21023cc55)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:53:07.614989 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2487851-67ee-415c-8f49-890b74500762 None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:53:07.618979 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 65/129] 10.222.0.50 () {56 vars in 1177 bytes} [Tue Sep 13 15:53:07 2022] GET /identity/v3/users/5c4340db782249918182a2e21023cc55 => generated 275 bytes in 197 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:53:07.620560 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2487851-67ee-415c-8f49-890b74500762 None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:53:07.648497 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6457d4a-9111-40d3-bc7b-21cec586f642 None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:53:07.685976 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe14a81f-0506-4941-a982-126685f9ced9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:53:07.686617 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe14a81f-0506-4941-a982-126685f9ced9 None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:07.686910 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe14a81f-0506-4941-a982-126685f9ced9 None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:07.687202 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe14a81f-0506-4941-a982-126685f9ced9 None admin] PATH_INFO: `/v3/projects/423edf83b2b24540924e990bb324af1e` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:07.688351 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe14a81f-0506-4941-a982-126685f9ced9 None admin] RBAC: Authorizing `identity:get_project(project_id=423edf83b2b24540924e990bb324af1e)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:53:07.690249 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe14a81f-0506-4941-a982-126685f9ced9 None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:53:07.691998 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 65/130] 10.222.0.50 () {56 vars in 1186 bytes} [Tue Sep 13 15:53:07 2022] GET /identity/v3/projects/423edf83b2b24540924e990bb324af1e => generated 298 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:53:07.696814 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe14a81f-0506-4941-a982-126685f9ced9 None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:53:07.739150 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2487851-67ee-415c-8f49-890b74500762 None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:53:07.753637 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:53:07.754623 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:53:07.778827 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-859fc1ad-79bb-41a9-90fd-21b1cf365e31 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:53:07.779817 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-859fc1ad-79bb-41a9-90fd-21b1cf365e31 None admin] REQUEST_METHOD: `PUT` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:07.780309 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-859fc1ad-79bb-41a9-90fd-21b1cf365e31 None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:07.780766 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-859fc1ad-79bb-41a9-90fd-21b1cf365e31 None admin] PATH_INFO: `/v3/projects/1df61ac8263c4c1e9847d7bdfbcdc627/users/5c4340db782249918182a2e21023cc55/roles/15db4aaca3264ae29bde38903ba43471` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:07.783066 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-859fc1ad-79bb-41a9-90fd-21b1cf365e31 None admin] RBAC: Authorizing `identity:create_grant(project_id=1df61ac8263c4c1e9847d7bdfbcdc627, user_id=5c4340db782249918182a2e21023cc55, role_id=15db4aaca3264ae29bde38903ba43471)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:53:07.784702 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-859fc1ad-79bb-41a9-90fd-21b1cf365e31 None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:53:07.802805 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe14a81f-0506-4941-a982-126685f9ced9 None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:53:07.803635 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 66/131] 10.222.0.50 () {58 vars in 1439 bytes} [Tue Sep 13 15:53:07 2022] PUT /identity/v3/projects/1df61ac8263c4c1e9847d7bdfbcdc627/users/5c4340db782249918182a2e21023cc55/roles/15db4aaca3264ae29bde38903ba43471 => generated 0 bytes in 184 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:53:07.804654 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-859fc1ad-79bb-41a9-90fd-21b1cf365e31 None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:53:07.905225 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54cf6d4c-7ab8-423d-9a01-cefba24baa2e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:53:07.905796 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54cf6d4c-7ab8-423d-9a01-cefba24baa2e None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:07.906077 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54cf6d4c-7ab8-423d-9a01-cefba24baa2e None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:07.906353 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54cf6d4c-7ab8-423d-9a01-cefba24baa2e None admin] PATH_INFO: `/v3/users/e0a915a0eb3c42c7af97dcc7eed5b42f` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:07.911522 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54cf6d4c-7ab8-423d-9a01-cefba24baa2e None admin] RBAC: Authorizing `identity:get_user(user_id=e0a915a0eb3c42c7af97dcc7eed5b42f)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:53:07.913016 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54cf6d4c-7ab8-423d-9a01-cefba24baa2e None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:53:07.915059 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 66/132] 10.222.0.50 () {56 vars in 1177 bytes} [Tue Sep 13 15:53:07 2022] GET /identity/v3/users/e0a915a0eb3c42c7af97dcc7eed5b42f => generated 247 bytes in 222 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:53:07.915843 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54cf6d4c-7ab8-423d-9a01-cefba24baa2e None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:53:07.938402 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-859fc1ad-79bb-41a9-90fd-21b1cf365e31 None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:53:07.981081 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61badcc8-d409-458d-9785-8ec45fc35c9d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:53:07.982246 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61badcc8-d409-458d-9785-8ec45fc35c9d None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:07.982757 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61badcc8-d409-458d-9785-8ec45fc35c9d None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:07.983258 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61badcc8-d409-458d-9785-8ec45fc35c9d None admin] PATH_INFO: `/v3/projects/423edf83b2b24540924e990bb324af1e` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:07.984724 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61badcc8-d409-458d-9785-8ec45fc35c9d None admin] RBAC: Authorizing `identity:get_project(project_id=423edf83b2b24540924e990bb324af1e)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:53:07.987052 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61badcc8-d409-458d-9785-8ec45fc35c9d None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:53:07.989003 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 67/133] 10.222.0.50 () {56 vars in 1186 bytes} [Tue Sep 13 15:53:07 2022] GET /identity/v3/projects/423edf83b2b24540924e990bb324af1e => generated 298 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:53:07.990160 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61badcc8-d409-458d-9785-8ec45fc35c9d None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:53:08.046497 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:53:08.133153 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61badcc8-d409-458d-9785-8ec45fc35c9d None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:53:08.133548 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54cf6d4c-7ab8-423d-9a01-cefba24baa2e None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:53:08.162350 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:53:08.174756 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acb0568b-6efa-4f65-aa3d-34388f7a2c79 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:53:08.175433 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acb0568b-6efa-4f65-aa3d-34388f7a2c79 None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:08.175856 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acb0568b-6efa-4f65-aa3d-34388f7a2c79 None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:08.176241 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acb0568b-6efa-4f65-aa3d-34388f7a2c79 None admin] PATH_INFO: `/v3/projects/423edf83b2b24540924e990bb324af1e` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:08.176860 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a8a1cde-7108-4a4a-bedd-eb3d886b9e0a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:53:08.177737 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a8a1cde-7108-4a4a-bedd-eb3d886b9e0a None admin] REQUEST_METHOD: `PUT` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:08.178166 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a8a1cde-7108-4a4a-bedd-eb3d886b9e0a None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:08.178576 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a8a1cde-7108-4a4a-bedd-eb3d886b9e0a None admin] PATH_INFO: `/v3/projects/423edf83b2b24540924e990bb324af1e/users/5c4340db782249918182a2e21023cc55/roles/15db4aaca3264ae29bde38903ba43471` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:08.178930 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-acb0568b-6efa-4f65-aa3d-34388f7a2c79 None admin] RBAC: Authorizing `identity:get_project(project_id=423edf83b2b24540924e990bb324af1e)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:53:08.180798 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a8a1cde-7108-4a4a-bedd-eb3d886b9e0a None admin] RBAC: Authorizing `identity:create_grant(project_id=423edf83b2b24540924e990bb324af1e, user_id=5c4340db782249918182a2e21023cc55, role_id=15db4aaca3264ae29bde38903ba43471)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:53:08.181191 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-acb0568b-6efa-4f65-aa3d-34388f7a2c79 None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:53:08.183526 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a8a1cde-7108-4a4a-bedd-eb3d886b9e0a None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:53:08.184359 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 68/134] 10.222.0.50 () {56 vars in 1186 bytes} [Tue Sep 13 15:53:07 2022] GET /identity/v3/projects/423edf83b2b24540924e990bb324af1e => generated 298 bytes in 194 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:53:08.185179 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acb0568b-6efa-4f65-aa3d-34388f7a2c79 None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:53:08.201306 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 67/135] 10.222.0.50 () {58 vars in 1439 bytes} [Tue Sep 13 15:53:07 2022] PUT /identity/v3/projects/423edf83b2b24540924e990bb324af1e/users/5c4340db782249918182a2e21023cc55/roles/15db4aaca3264ae29bde38903ba43471 => generated 0 bytes in 286 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:53:08.203109 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a8a1cde-7108-4a4a-bedd-eb3d886b9e0a None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:53:08.349642 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acb0568b-6efa-4f65-aa3d-34388f7a2c79 None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:53:08.363699 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:53:08.395954 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fa9469a-43b6-4397-a75d-cc5d6a4c2f0a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:53:08.396645 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fa9469a-43b6-4397-a75d-cc5d6a4c2f0a None admin] REQUEST_METHOD: `PUT` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:08.396929 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fa9469a-43b6-4397-a75d-cc5d6a4c2f0a None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:08.397209 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fa9469a-43b6-4397-a75d-cc5d6a4c2f0a None admin] PATH_INFO: `/v3/projects/423edf83b2b24540924e990bb324af1e/users/5c4340db782249918182a2e21023cc55/roles/f74e3f6c20bb426ca030f211c3a9aef5` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:08.402530 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2fa9469a-43b6-4397-a75d-cc5d6a4c2f0a None admin] RBAC: Authorizing `identity:create_grant(project_id=423edf83b2b24540924e990bb324af1e, user_id=5c4340db782249918182a2e21023cc55, role_id=f74e3f6c20bb426ca030f211c3a9aef5)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:53:08.404070 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2fa9469a-43b6-4397-a75d-cc5d6a4c2f0a None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:53:08.421950 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a8a1cde-7108-4a4a-bedd-eb3d886b9e0a None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:53:08.430591 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 69/136] 10.222.0.50 () {58 vars in 1439 bytes} [Tue Sep 13 15:53:08 2022] PUT /identity/v3/projects/423edf83b2b24540924e990bb324af1e/users/5c4340db782249918182a2e21023cc55/roles/f74e3f6c20bb426ca030f211c3a9aef5 => generated 0 bytes in 245 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:53:08.432203 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a293673-ad5d-4c42-90dd-868b1a574cdc None None] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:08.433704 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a293673-ad5d-4c42-90dd-868b1a574cdc None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:08.434011 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a293673-ad5d-4c42-90dd-868b1a574cdc None None] PATH_INFO: `/` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:08.435753 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 70/137] 10.222.0.50 () {54 vars in 927 bytes} [Tue Sep 13 15:53:08 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:53:08.442064 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-569c263f-9f5c-4b3b-8e53-751d3e0290a2 None None] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:08.442387 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-569c263f-9f5c-4b3b-8e53-751d3e0290a2 None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:08.442622 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-569c263f-9f5c-4b3b-8e53-751d3e0290a2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:08.468471 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.auth.core [None req-569c263f-9f5c-4b3b-8e53-751d3e0290a2 None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81437) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:53:08.469727 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-569c263f-9f5c-4b3b-8e53-751d3e0290a2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:53:08.508497 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5a7819f-fb2e-4dca-8c55-3724ca348b7b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:53:08.509195 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5a7819f-fb2e-4dca-8c55-3724ca348b7b None admin] REQUEST_METHOD: `PUT` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:08.509481 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5a7819f-fb2e-4dca-8c55-3724ca348b7b None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:08.509760 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5a7819f-fb2e-4dca-8c55-3724ca348b7b None admin] PATH_INFO: `/v3/projects/423edf83b2b24540924e990bb324af1e/users/e0a915a0eb3c42c7af97dcc7eed5b42f/roles/031d4e1a23e34f71858fc4a6dfbd4ccb` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:08.513958 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5a7819f-fb2e-4dca-8c55-3724ca348b7b None admin] RBAC: Authorizing `identity:create_grant(project_id=423edf83b2b24540924e990bb324af1e, user_id=e0a915a0eb3c42c7af97dcc7eed5b42f, role_id=031d4e1a23e34f71858fc4a6dfbd4ccb)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:53:08.515464 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5a7819f-fb2e-4dca-8c55-3724ca348b7b None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:53:08.540272 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 68/138] 10.222.0.50 () {58 vars in 1439 bytes} [Tue Sep 13 15:53:08 2022] PUT /identity/v3/projects/423edf83b2b24540924e990bb324af1e/users/e0a915a0eb3c42c7af97dcc7eed5b42f/roles/031d4e1a23e34f71858fc4a6dfbd4ccb => generated 0 bytes in 338 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:53:08.585038 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 71/139] 10.222.0.50 () {58 vars in 1024 bytes} [Tue Sep 13 15:53:08 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 144 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 13 15:53:08.883659 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5a7819f-fb2e-4dca-8c55-3724ca348b7b None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:53:08.932706 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:53:09.004592 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5a7819f-fb2e-4dca-8c55-3724ca348b7b None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:53:09.050492 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9544212b-68dd-4a97-95be-dfcaf759d5e2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:53:09.051229 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9544212b-68dd-4a97-95be-dfcaf759d5e2 None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:09.051515 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9544212b-68dd-4a97-95be-dfcaf759d5e2 None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:09.051800 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9544212b-68dd-4a97-95be-dfcaf759d5e2 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:09.053038 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9544212b-68dd-4a97-95be-dfcaf759d5e2 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:53:09.054504 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9544212b-68dd-4a97-95be-dfcaf759d5e2 None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:53:09.056134 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 69/140] 10.222.0.50 () {56 vars in 1108 bytes} [Tue Sep 13 15:53:08 2022] GET /identity/v3/domains/default => generated 199 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:53:09.062960 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-569c263f-9f5c-4b3b-8e53-751d3e0290a2 None None] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:53:09.191288 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:53:09.195497 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-569c263f-9f5c-4b3b-8e53-751d3e0290a2 None None] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:53:09.241135 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a07c29c8-9ae0-4be5-aaf9-99111ab2c97d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:53:09.241828 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a07c29c8-9ae0-4be5-aaf9-99111ab2c97d None admin] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:09.242114 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a07c29c8-9ae0-4be5-aaf9-99111ab2c97d None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:09.242386 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a07c29c8-9ae0-4be5-aaf9-99111ab2c97d None admin] PATH_INFO: `/v3/users` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:09.243074 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a07c29c8-9ae0-4be5-aaf9-99111ab2c97d None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:53:09.244612 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a07c29c8-9ae0-4be5-aaf9-99111ab2c97d None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:53:09.278824 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 72/141] 10.222.0.50 () {60 vars in 1132 bytes} [Tue Sep 13 15:53:09 2022] POST /identity/v3/users => generated 283 bytes in 216 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 15:53:09.657824 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a79973d-27fc-46bb-94ce-b6a2603268f8 None None] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:09.658338 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a79973d-27fc-46bb-94ce-b6a2603268f8 None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:09.658437 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a79973d-27fc-46bb-94ce-b6a2603268f8 None None] PATH_INFO: `/` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:09.659934 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 70/142] 10.222.0.50 () {54 vars in 927 bytes} [Tue Sep 13 15:53:09 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:53:09.665503 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e57fade-379e-43a5-ac84-cf6f99dd0991 None None] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:09.666341 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e57fade-379e-43a5-ac84-cf6f99dd0991 None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:09.667534 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e57fade-379e-43a5-ac84-cf6f99dd0991 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:09.696426 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.auth.core [None req-2e57fade-379e-43a5-ac84-cf6f99dd0991 None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81437) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:53:09.698066 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-2e57fade-379e-43a5-ac84-cf6f99dd0991 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:53:09.787784 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 73/143] 10.222.0.50 () {58 vars in 1024 bytes} [Tue Sep 13 15:53:09 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 123 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 13 15:53:10.340521 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a79973d-27fc-46bb-94ce-b6a2603268f8 None None] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:53:10.465213 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a79973d-27fc-46bb-94ce-b6a2603268f8 None None] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:53:10.478666 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:53:10.502147 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f764f23d-69f9-4313-9f77-e068397a3cbe None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:53:10.502798 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f764f23d-69f9-4313-9f77-e068397a3cbe None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:10.503111 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f764f23d-69f9-4313-9f77-e068397a3cbe None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:10.503450 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f764f23d-69f9-4313-9f77-e068397a3cbe None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:10.516740 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f764f23d-69f9-4313-9f77-e068397a3cbe None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 13 15:53:10.521629 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application [None req-f764f23d-69f9-4313-9f77-e068397a3cbe None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Sep 13 15:53:10.521629 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:53:10.521629 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:53:10.521629 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:53:10.521629 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:53:10.521629 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:53:10.521629 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:53:10.521629 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:53:10.521629 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:53:10.521629 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:53:10.521629 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:53:10.521629 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:53:10.521629 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 13 15:53:10.521629 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 13 15:53:10.521629 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 13 15:53:10.521629 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 13 15:53:10.522466 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 13 15:53:10.522466 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 13 15:53:10.522466 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 13 15:53:10.522466 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 13 15:53:10.522466 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:53:10.522466 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:53:10.522466 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:53:10.522466 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:53:10.522466 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:53:10.522466 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:53:10.522466 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:53:10.522466 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application with Lock( Sep 13 15:53:10.522466 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:53:10.522466 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:53:10.522466 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:53:10.522466 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:53:10.523916 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:53:10.523916 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:53:10.523916 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:53:10.523916 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:53:10.523916 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 13 15:53:10.523916 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 13 15:53:10.523916 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 13 15:53:10.523916 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 13 15:53:10.523916 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 13 15:53:10.523916 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 13 15:53:10.523916 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Sep 13 15:53:10.523916 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Sep 13 15:53:10.523916 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 71/144] 10.222.0.50 () {56 vars in 1099 bytes} [Tue Sep 13 15:53:10 2022] GET /identity/v3/roles/reader => generated 84 bytes in 183 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:53:10.530994 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e57fade-379e-43a5-ac84-cf6f99dd0991 None None] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:53:10.629057 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:53:10.645607 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e57fade-379e-43a5-ac84-cf6f99dd0991 None None] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:53:10.681814 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fdbf0b6-5372-4128-83b8-24d986b4f50d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:53:10.683295 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fdbf0b6-5372-4128-83b8-24d986b4f50d None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:10.683578 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fdbf0b6-5372-4128-83b8-24d986b4f50d None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:10.683850 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fdbf0b6-5372-4128-83b8-24d986b4f50d None admin] PATH_INFO: `/v3/roles` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:10.684523 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fdbf0b6-5372-4128-83b8-24d986b4f50d None admin] RBAC: Adding query filter params (name=reader) {{(pid=81437) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:53:10.684826 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fdbf0b6-5372-4128-83b8-24d986b4f50d None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:53:10.686229 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fdbf0b6-5372-4128-83b8-24d986b4f50d None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:53:10.696159 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 74/145] 10.222.0.50 () {56 vars in 1101 bytes} [Tue Sep 13 15:53:10 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:53:10.702010 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f764f23d-69f9-4313-9f77-e068397a3cbe None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:53:10.823561 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f764f23d-69f9-4313-9f77-e068397a3cbe None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:53:10.870318 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed322b2c-265e-4a1a-88e2-35b43cea6ad3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:53:10.871072 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed322b2c-265e-4a1a-88e2-35b43cea6ad3 None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:10.871360 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed322b2c-265e-4a1a-88e2-35b43cea6ad3 None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:10.871637 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed322b2c-265e-4a1a-88e2-35b43cea6ad3 None admin] PATH_INFO: `/v3/users/83ce764787ea4cc389a55bbb8d35c45f` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:10.872797 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed322b2c-265e-4a1a-88e2-35b43cea6ad3 None admin] RBAC: Authorizing `identity:get_user(user_id=83ce764787ea4cc389a55bbb8d35c45f)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:53:10.874273 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed322b2c-265e-4a1a-88e2-35b43cea6ad3 None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:53:10.876219 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 72/146] 10.222.0.50 () {56 vars in 1177 bytes} [Tue Sep 13 15:53:10 2022] GET /identity/v3/users/83ce764787ea4cc389a55bbb8d35c45f => generated 289 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:53:10.881701 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fdbf0b6-5372-4128-83b8-24d986b4f50d None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:53:11.003670 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fdbf0b6-5372-4128-83b8-24d986b4f50d None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:53:11.046167 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-374931e5-6905-494a-8ec2-ccce45fab3cd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:53:11.046829 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-374931e5-6905-494a-8ec2-ccce45fab3cd None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:11.047134 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-374931e5-6905-494a-8ec2-ccce45fab3cd None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:11.047408 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-374931e5-6905-494a-8ec2-ccce45fab3cd None admin] PATH_INFO: `/v3/projects/423edf83b2b24540924e990bb324af1e` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:11.050167 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-374931e5-6905-494a-8ec2-ccce45fab3cd None admin] RBAC: Authorizing `identity:get_project(project_id=423edf83b2b24540924e990bb324af1e)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:53:11.051735 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-374931e5-6905-494a-8ec2-ccce45fab3cd None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:53:11.053700 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 75/147] 10.222.0.50 () {56 vars in 1186 bytes} [Tue Sep 13 15:53:10 2022] GET /identity/v3/projects/423edf83b2b24540924e990bb324af1e => generated 298 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:53:11.060708 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed322b2c-265e-4a1a-88e2-35b43cea6ad3 None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:53:11.130027 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:53:11.198542 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed322b2c-265e-4a1a-88e2-35b43cea6ad3 None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:53:11.234190 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a82eac37-0746-43a1-b496-ec048810ed54 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:53:11.235267 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a82eac37-0746-43a1-b496-ec048810ed54 None admin] REQUEST_METHOD: `PUT` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:11.235775 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a82eac37-0746-43a1-b496-ec048810ed54 None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:11.236259 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a82eac37-0746-43a1-b496-ec048810ed54 None admin] PATH_INFO: `/v3/projects/423edf83b2b24540924e990bb324af1e/users/83ce764787ea4cc389a55bbb8d35c45f/roles/b2d1670446a54085ad2bdb935f003d25` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:11.238440 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a82eac37-0746-43a1-b496-ec048810ed54 None admin] RBAC: Authorizing `identity:create_grant(project_id=423edf83b2b24540924e990bb324af1e, user_id=83ce764787ea4cc389a55bbb8d35c45f, role_id=b2d1670446a54085ad2bdb935f003d25)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:53:11.240148 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a82eac37-0746-43a1-b496-ec048810ed54 None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:53:11.258291 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 73/148] 10.222.0.50 () {58 vars in 1439 bytes} [Tue Sep 13 15:53:11 2022] PUT /identity/v3/projects/423edf83b2b24540924e990bb324af1e/users/83ce764787ea4cc389a55bbb8d35c45f/roles/b2d1670446a54085ad2bdb935f003d25 => generated 0 bytes in 198 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:53:11.565192 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7846c76b-4f72-4040-8e7d-e245775ce296 None None] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:11.565381 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7846c76b-4f72-4040-8e7d-e245775ce296 None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:11.565652 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7846c76b-4f72-4040-8e7d-e245775ce296 None None] PATH_INFO: `/` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:11.567483 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 76/149] 10.222.0.50 () {54 vars in 927 bytes} [Tue Sep 13 15:53:11 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:53:11.580679 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-582931cb-a83f-4a43-9a5d-41b5c19a9571 None None] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:11.581205 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-582931cb-a83f-4a43-9a5d-41b5c19a9571 None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:11.581647 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-582931cb-a83f-4a43-9a5d-41b5c19a9571 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:11.607960 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.auth.core [None req-582931cb-a83f-4a43-9a5d-41b5c19a9571 None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81438) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:53:11.609460 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-582931cb-a83f-4a43-9a5d-41b5c19a9571 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:53:11.712877 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 74/150] 10.222.0.50 () {58 vars in 1024 bytes} [Tue Sep 13 15:53:11 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 133 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 13 15:53:11.829439 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-722afa6f-60b0-422b-b85c-c0d9b26248a3 None None] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:11.829933 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-722afa6f-60b0-422b-b85c-c0d9b26248a3 None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:11.830028 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-722afa6f-60b0-422b-b85c-c0d9b26248a3 None None] PATH_INFO: `/` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:11.831535 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 77/151] 10.222.0.50 () {54 vars in 927 bytes} [Tue Sep 13 15:53:11 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:53:11.837167 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c45d1bc-643a-42d2-be92-7deabbfed848 None None] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:11.837466 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c45d1bc-643a-42d2-be92-7deabbfed848 None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:11.837743 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c45d1bc-643a-42d2-be92-7deabbfed848 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:11.859396 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.auth.core [None req-5c45d1bc-643a-42d2-be92-7deabbfed848 None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81438) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:53:11.860585 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-5c45d1bc-643a-42d2-be92-7deabbfed848 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:53:11.888415 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:53:11.940125 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 75/152] 10.222.0.50 () {58 vars in 1024 bytes} [Tue Sep 13 15:53:11 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 13 15:53:12.080587 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22b4eeb8-a1c9-4090-8c6b-b0f58d6ba217 None None] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:12.080787 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22b4eeb8-a1c9-4090-8c6b-b0f58d6ba217 None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:12.081056 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22b4eeb8-a1c9-4090-8c6b-b0f58d6ba217 None None] PATH_INFO: `/` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:12.082851 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 78/153] 10.222.0.50 () {54 vars in 927 bytes} [Tue Sep 13 15:53:12 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:53:12.096958 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c00cbe8-ec00-4f0a-8fde-bca01112a93c None None] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:12.097277 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c00cbe8-ec00-4f0a-8fde-bca01112a93c None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:12.097518 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c00cbe8-ec00-4f0a-8fde-bca01112a93c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:12.111120 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c904e2c-0553-4255-bc9c-67a1de3de708 None None] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:12.111433 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c904e2c-0553-4255-bc9c-67a1de3de708 None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:12.111678 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c904e2c-0553-4255-bc9c-67a1de3de708 None None] PATH_INFO: `/` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:12.113280 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 79/154] 10.222.0.50 () {54 vars in 927 bytes} [Tue Sep 13 15:53:12 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:53:12.125009 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.auth.core [None req-5c00cbe8-ec00-4f0a-8fde-bca01112a93c None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81438) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:53:12.126205 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-5c00cbe8-ec00-4f0a-8fde-bca01112a93c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:53:12.133910 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe85da54-81e4-4e3d-bac9-7388d81054d2 None None] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:12.134186 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe85da54-81e4-4e3d-bac9-7388d81054d2 None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:12.134468 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe85da54-81e4-4e3d-bac9-7388d81054d2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:12.161394 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.auth.core [None req-fe85da54-81e4-4e3d-bac9-7388d81054d2 None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81437) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:53:12.162602 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-fe85da54-81e4-4e3d-bac9-7388d81054d2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:53:12.183282 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:53:12.245185 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 76/155] 10.222.0.50 () {58 vars in 1024 bytes} [Tue Sep 13 15:53:12 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 149 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 13 15:53:12.283072 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 80/156] 10.222.0.50 () {58 vars in 1024 bytes} [Tue Sep 13 15:53:12 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 150 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 13 15:53:12.325692 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c00cbe8-ec00-4f0a-8fde-bca01112a93c None None] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:53:12.425449 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:53:12.436315 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c00cbe8-ec00-4f0a-8fde-bca01112a93c None None] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:53:12.477780 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be184a9a-9029-48aa-9487-f52a080b7ef9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:53:12.478429 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be184a9a-9029-48aa-9487-f52a080b7ef9 None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:12.478766 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be184a9a-9029-48aa-9487-f52a080b7ef9 None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:12.479078 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be184a9a-9029-48aa-9487-f52a080b7ef9 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:12.492404 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be184a9a-9029-48aa-9487-f52a080b7ef9 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 13 15:53:12.498403 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application [None req-be184a9a-9029-48aa-9487-f52a080b7ef9 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 13 15:53:12.498403 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:53:12.498403 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:53:12.498403 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:53:12.498403 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:53:12.498403 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:53:12.498403 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:53:12.498403 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:53:12.498403 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:53:12.498403 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:53:12.498403 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:53:12.498403 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:53:12.498403 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 13 15:53:12.498403 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 13 15:53:12.498403 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 13 15:53:12.498403 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 13 15:53:12.499325 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 13 15:53:12.499325 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 13 15:53:12.499325 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 13 15:53:12.499325 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 13 15:53:12.499325 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:53:12.499325 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:53:12.499325 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:53:12.499325 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:53:12.499325 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:53:12.499325 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:53:12.499325 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:53:12.499325 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application with Lock( Sep 13 15:53:12.499325 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:53:12.499325 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:53:12.499325 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:53:12.499325 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:53:12.500117 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:53:12.500117 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:53:12.500117 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:53:12.500117 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:53:12.500117 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 13 15:53:12.500117 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 13 15:53:12.500117 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 13 15:53:12.500117 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 13 15:53:12.500117 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 13 15:53:12.500117 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 13 15:53:12.500117 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 13 15:53:12.500117 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Sep 13 15:53:12.500117 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 77/157] 10.222.0.50 () {56 vars in 1099 bytes} [Tue Sep 13 15:53:12 2022] GET /identity/v3/roles/member => generated 84 bytes in 174 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:53:12.506636 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe85da54-81e4-4e3d-bac9-7388d81054d2 None None] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:53:12.617903 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be184a9a-9029-48aa-9487-f52a080b7ef9 None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:53:12.664537 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe85da54-81e4-4e3d-bac9-7388d81054d2 None None] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:53:12.742755 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db086df1-ff2c-436a-9b25-0aaae385c6b6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:53:12.745757 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db086df1-ff2c-436a-9b25-0aaae385c6b6 None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:12.746226 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db086df1-ff2c-436a-9b25-0aaae385c6b6 None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:12.746697 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db086df1-ff2c-436a-9b25-0aaae385c6b6 None admin] PATH_INFO: `/v3/roles` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:12.747632 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db086df1-ff2c-436a-9b25-0aaae385c6b6 None admin] RBAC: Adding query filter params (name=member) {{(pid=81437) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:53:12.748100 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db086df1-ff2c-436a-9b25-0aaae385c6b6 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:53:12.749656 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db086df1-ff2c-436a-9b25-0aaae385c6b6 None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:53:12.762925 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 81/158] 10.222.0.50 () {56 vars in 1101 bytes} [Tue Sep 13 15:53:12 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 252 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:53:12.772160 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db086df1-ff2c-436a-9b25-0aaae385c6b6 None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:53:12.802929 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:53:12.842652 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be184a9a-9029-48aa-9487-f52a080b7ef9 None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:53:12.922951 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f55254f5-e816-4f45-bca2-973094dcc23c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:53:12.922951 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f55254f5-e816-4f45-bca2-973094dcc23c None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:12.922951 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f55254f5-e816-4f45-bca2-973094dcc23c None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:12.922951 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f55254f5-e816-4f45-bca2-973094dcc23c None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:12.925533 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db086df1-ff2c-436a-9b25-0aaae385c6b6 None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:53:12.952576 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f55254f5-e816-4f45-bca2-973094dcc23c None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 13 15:53:12.963227 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application [None req-f55254f5-e816-4f45-bca2-973094dcc23c None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 13 15:53:12.963227 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:53:12.963227 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:53:12.963227 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:53:12.963227 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:53:12.963227 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:53:12.963227 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:53:12.963227 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:53:12.963227 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:53:12.963227 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:53:12.963227 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:53:12.963227 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:53:12.963227 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 13 15:53:12.963227 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 13 15:53:12.963227 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 13 15:53:12.963227 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 13 15:53:12.965120 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 13 15:53:12.965120 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 13 15:53:12.965120 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 13 15:53:12.965120 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 13 15:53:12.965120 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:53:12.965120 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:53:12.965120 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:53:12.965120 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:53:12.965120 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:53:12.965120 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:53:12.965120 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:53:12.965120 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application with Lock( Sep 13 15:53:12.965120 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:53:12.965120 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:53:12.965120 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:53:12.965120 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:53:12.965986 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:53:12.965986 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:53:12.965986 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:53:12.965986 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:53:12.965986 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 13 15:53:12.965986 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 13 15:53:12.965986 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 13 15:53:12.965986 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 13 15:53:12.965986 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 13 15:53:12.965986 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 13 15:53:12.965986 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 13 15:53:12.965986 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Sep 13 15:53:12.965986 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 78/159] 10.222.0.50 () {56 vars in 1114 bytes} [Tue Sep 13 15:53:12 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 341 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 13 15:53:12.967602 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f55254f5-e816-4f45-bca2-973094dcc23c None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:53:12.981949 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa70c163-5025-418b-abf0-ae81b9898b4b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:53:12.983028 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa70c163-5025-418b-abf0-ae81b9898b4b None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:12.983458 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa70c163-5025-418b-abf0-ae81b9898b4b None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:12.983881 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa70c163-5025-418b-abf0-ae81b9898b4b None admin] PATH_INFO: `/v3/users/5c4340db782249918182a2e21023cc55` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:12.987713 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa70c163-5025-418b-abf0-ae81b9898b4b None admin] RBAC: Authorizing `identity:get_user(user_id=5c4340db782249918182a2e21023cc55)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:53:12.989460 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa70c163-5025-418b-abf0-ae81b9898b4b None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:53:12.991458 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 82/160] 10.222.0.50 () {56 vars in 1177 bytes} [Tue Sep 13 15:53:12 2022] GET /identity/v3/users/5c4340db782249918182a2e21023cc55 => generated 275 bytes in 220 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 13 15:53:12.992577 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa70c163-5025-418b-abf0-ae81b9898b4b None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:53:12.998481 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:53:13.160829 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:53:13.164964 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f55254f5-e816-4f45-bca2-973094dcc23c None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:53:13.196176 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa70c163-5025-418b-abf0-ae81b9898b4b None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:53:13.228993 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d41b37a9-b74f-495a-bcfd-bb1fadcdbf2a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:53:13.229690 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d41b37a9-b74f-495a-bcfd-bb1fadcdbf2a None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:13.230076 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d41b37a9-b74f-495a-bcfd-bb1fadcdbf2a None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:13.230356 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d41b37a9-b74f-495a-bcfd-bb1fadcdbf2a None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:13.253471 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d41b37a9-b74f-495a-bcfd-bb1fadcdbf2a None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 13 15:53:13.259184 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application [None req-d41b37a9-b74f-495a-bcfd-bb1fadcdbf2a None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 13 15:53:13.259184 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:53:13.259184 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:53:13.259184 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:53:13.259184 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:53:13.259184 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:53:13.259184 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:53:13.259184 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:53:13.259184 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:53:13.259184 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:53:13.259184 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:53:13.259184 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:53:13.259184 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 13 15:53:13.259184 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 13 15:53:13.259184 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 13 15:53:13.259184 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 13 15:53:13.260110 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 13 15:53:13.260110 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 13 15:53:13.260110 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 13 15:53:13.260110 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 13 15:53:13.260110 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:53:13.260110 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:53:13.260110 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:53:13.260110 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:53:13.260110 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:53:13.260110 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:53:13.260110 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:53:13.260110 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application with Lock( Sep 13 15:53:13.260110 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:53:13.260110 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:53:13.260110 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:53:13.260110 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:53:13.261522 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:53:13.261522 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:53:13.261522 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:53:13.261522 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:53:13.261522 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 13 15:53:13.261522 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 13 15:53:13.261522 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 13 15:53:13.261522 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 13 15:53:13.261522 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 13 15:53:13.261522 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 13 15:53:13.261522 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 13 15:53:13.261522 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Sep 13 15:53:13.262805 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d715a5c-6954-440b-b9ea-1da444fbf724 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:53:13.262805 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d715a5c-6954-440b-b9ea-1da444fbf724 None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:13.263134 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 79/161] 10.222.0.50 () {56 vars in 1096 bytes} [Tue Sep 13 15:53:12 2022] GET /identity/v3/roles/admin => generated 83 bytes in 301 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:53:13.263287 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d715a5c-6954-440b-b9ea-1da444fbf724 None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:13.263287 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d715a5c-6954-440b-b9ea-1da444fbf724 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:13.266538 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d41b37a9-b74f-495a-bcfd-bb1fadcdbf2a None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:53:13.301116 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d715a5c-6954-440b-b9ea-1da444fbf724 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 13 15:53:13.308437 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application [None req-6d715a5c-6954-440b-b9ea-1da444fbf724 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 13 15:53:13.308437 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:53:13.308437 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:53:13.308437 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:53:13.308437 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:53:13.308437 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:53:13.308437 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:53:13.308437 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:53:13.308437 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:53:13.308437 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:53:13.308437 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:53:13.308437 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:53:13.308437 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 13 15:53:13.308437 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 13 15:53:13.308437 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 13 15:53:13.308437 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 13 15:53:13.310587 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 13 15:53:13.310587 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 13 15:53:13.310587 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 13 15:53:13.310587 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 13 15:53:13.310587 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:53:13.310587 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:53:13.310587 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:53:13.310587 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:53:13.310587 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:53:13.310587 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:53:13.310587 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:53:13.310587 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application with Lock( Sep 13 15:53:13.310587 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:53:13.310587 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:53:13.310587 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:53:13.310587 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:53:13.312761 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:53:13.312761 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:53:13.312761 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:53:13.312761 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:53:13.312761 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 13 15:53:13.312761 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 13 15:53:13.312761 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 13 15:53:13.312761 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 13 15:53:13.312761 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 13 15:53:13.312761 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 13 15:53:13.312761 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 13 15:53:13.312761 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application Sep 13 15:53:13.312761 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 83/162] 10.222.0.50 () {56 vars in 1099 bytes} [Tue Sep 13 15:53:12 2022] GET /identity/v3/roles/member => generated 84 bytes in 317 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:53:13.314834 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d715a5c-6954-440b-b9ea-1da444fbf724 None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:53:13.403331 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:53:13.495641 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d41b37a9-b74f-495a-bcfd-bb1fadcdbf2a None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:53:13.540239 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d715a5c-6954-440b-b9ea-1da444fbf724 None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:53:13.560561 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ae3fb0f-38f3-41d5-a7c8-6c8ca58b144b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:53:13.561199 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ae3fb0f-38f3-41d5-a7c8-6c8ca58b144b None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:13.561484 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ae3fb0f-38f3-41d5-a7c8-6c8ca58b144b None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:13.561764 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ae3fb0f-38f3-41d5-a7c8-6c8ca58b144b None admin] PATH_INFO: `/v3/roles` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:13.562441 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ae3fb0f-38f3-41d5-a7c8-6c8ca58b144b None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81438) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:53:13.562748 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ae3fb0f-38f3-41d5-a7c8-6c8ca58b144b None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:53:13.564285 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ae3fb0f-38f3-41d5-a7c8-6c8ca58b144b None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:53:13.578392 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 80/163] 10.222.0.50 () {56 vars in 1111 bytes} [Tue Sep 13 15:53:13 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 318 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:53:13.579434 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ae3fb0f-38f3-41d5-a7c8-6c8ca58b144b None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:53:13.608963 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb73b8a4-17a3-4d96-ac44-ed3970f9fa9f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:53:13.609965 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb73b8a4-17a3-4d96-ac44-ed3970f9fa9f None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:13.610452 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb73b8a4-17a3-4d96-ac44-ed3970f9fa9f None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:13.610952 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb73b8a4-17a3-4d96-ac44-ed3970f9fa9f None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:13.618091 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:53:13.723294 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ae3fb0f-38f3-41d5-a7c8-6c8ca58b144b None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:53:13.739136 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb73b8a4-17a3-4d96-ac44-ed3970f9fa9f None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:53:13.740983 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb73b8a4-17a3-4d96-ac44-ed3970f9fa9f None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:53:13.766926 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:53:13.784169 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e4e283a-ef0f-4bf3-9ce2-d1b77e9e393b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:53:13.784813 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e4e283a-ef0f-4bf3-9ce2-d1b77e9e393b None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:13.785101 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e4e283a-ef0f-4bf3-9ce2-d1b77e9e393b None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:13.785378 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e4e283a-ef0f-4bf3-9ce2-d1b77e9e393b None admin] PATH_INFO: `/v3/roles` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:13.786059 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e4e283a-ef0f-4bf3-9ce2-d1b77e9e393b None admin] RBAC: Adding query filter params (name=admin) {{(pid=81438) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:53:13.786367 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e4e283a-ef0f-4bf3-9ce2-d1b77e9e393b None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:53:13.787783 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e4e283a-ef0f-4bf3-9ce2-d1b77e9e393b None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:53:13.801589 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 81/164] 10.222.0.50 () {56 vars in 1099 bytes} [Tue Sep 13 15:53:13 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 222 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:53:13.802287 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e4e283a-ef0f-4bf3-9ce2-d1b77e9e393b None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:53:13.938517 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 84/165] 10.222.0.50 () {58 vars in 1282 bytes} [Tue Sep 13 15:53:13 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 628 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 13 15:53:13.940200 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb73b8a4-17a3-4d96-ac44-ed3970f9fa9f None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:53:13.996680 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:53:14.032309 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e4e283a-ef0f-4bf3-9ce2-d1b77e9e393b None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:53:14.112882 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6dabefb-26a3-4bcb-9599-831e6e4fad6c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:53:14.113720 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6dabefb-26a3-4bcb-9599-831e6e4fad6c None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:14.114005 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6dabefb-26a3-4bcb-9599-831e6e4fad6c None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:14.114286 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6dabefb-26a3-4bcb-9599-831e6e4fad6c None admin] PATH_INFO: `/v3/roles` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:14.115369 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6dabefb-26a3-4bcb-9599-831e6e4fad6c None admin] RBAC: Adding query filter params (name=member) {{(pid=81438) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:53:14.115814 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6dabefb-26a3-4bcb-9599-831e6e4fad6c None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:53:14.117209 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6dabefb-26a3-4bcb-9599-831e6e4fad6c None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:53:14.130953 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb73b8a4-17a3-4d96-ac44-ed3970f9fa9f None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:53:14.140786 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 82/166] 10.222.0.50 () {56 vars in 1101 bytes} [Tue Sep 13 15:53:13 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 339 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:53:14.142112 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6dabefb-26a3-4bcb-9599-831e6e4fad6c None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:53:14.191331 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3aa1d82-8134-4c17-8407-67ffca311455 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:53:14.191982 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3aa1d82-8134-4c17-8407-67ffca311455 None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:14.192260 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3aa1d82-8134-4c17-8407-67ffca311455 None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:14.192534 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3aa1d82-8134-4c17-8407-67ffca311455 None admin] PATH_INFO: `/v3/users/5c4340db782249918182a2e21023cc55` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:14.193682 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3aa1d82-8134-4c17-8407-67ffca311455 None admin] RBAC: Authorizing `identity:get_user(user_id=5c4340db782249918182a2e21023cc55)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:53:14.195330 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3aa1d82-8134-4c17-8407-67ffca311455 None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:53:14.197354 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 85/167] 10.222.0.50 () {56 vars in 1177 bytes} [Tue Sep 13 15:53:13 2022] GET /identity/v3/users/5c4340db782249918182a2e21023cc55 => generated 275 bytes in 257 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:53:14.198853 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3aa1d82-8134-4c17-8407-67ffca311455 None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:53:14.280116 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:53:14.386371 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3aa1d82-8134-4c17-8407-67ffca311455 None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:53:14.388915 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:53:14.417736 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6dabefb-26a3-4bcb-9599-831e6e4fad6c None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:53:14.434693 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-360324fd-17ed-4373-a36a-66e6546825a3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:53:14.435756 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-360324fd-17ed-4373-a36a-66e6546825a3 None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:14.436264 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-360324fd-17ed-4373-a36a-66e6546825a3 None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:14.436743 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-360324fd-17ed-4373-a36a-66e6546825a3 None admin] PATH_INFO: `/v3/projects/1df61ac8263c4c1e9847d7bdfbcdc627` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:14.438135 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-360324fd-17ed-4373-a36a-66e6546825a3 None admin] RBAC: Authorizing `identity:get_project(project_id=1df61ac8263c4c1e9847d7bdfbcdc627)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:53:14.439853 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-360324fd-17ed-4373-a36a-66e6546825a3 None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:53:14.441726 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 86/168] 10.222.0.50 () {56 vars in 1186 bytes} [Tue Sep 13 15:53:14 2022] GET /identity/v3/projects/1df61ac8263c4c1e9847d7bdfbcdc627 => generated 312 bytes in 244 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:53:14.443184 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d9f698a-4d51-4876-b34d-f96721df1432 None None] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:14.443677 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d9f698a-4d51-4876-b34d-f96721df1432 None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:14.444147 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d9f698a-4d51-4876-b34d-f96721df1432 None None] PATH_INFO: `/` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:14.445778 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 87/169] 10.222.0.50 () {54 vars in 927 bytes} [Tue Sep 13 15:53:14 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:53:14.447065 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d9f698a-4d51-4876-b34d-f96721df1432 None None] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:53:14.511789 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd4e23b2-d6bc-4228-9ab2-dbf75aa2bd71 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:53:14.512617 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd4e23b2-d6bc-4228-9ab2-dbf75aa2bd71 None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:14.512915 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd4e23b2-d6bc-4228-9ab2-dbf75aa2bd71 None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:14.513196 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd4e23b2-d6bc-4228-9ab2-dbf75aa2bd71 None admin] PATH_INFO: `/v3/users/e0a915a0eb3c42c7af97dcc7eed5b42f` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:14.514415 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd4e23b2-d6bc-4228-9ab2-dbf75aa2bd71 None admin] RBAC: Authorizing `identity:get_user(user_id=e0a915a0eb3c42c7af97dcc7eed5b42f)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:53:14.516225 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd4e23b2-d6bc-4228-9ab2-dbf75aa2bd71 None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:53:14.517772 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 83/170] 10.222.0.50 () {56 vars in 1177 bytes} [Tue Sep 13 15:53:14 2022] GET /identity/v3/users/e0a915a0eb3c42c7af97dcc7eed5b42f => generated 247 bytes in 376 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:53:14.518404 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd4e23b2-d6bc-4228-9ab2-dbf75aa2bd71 None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:53:14.578861 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d9f698a-4d51-4876-b34d-f96721df1432 None None] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:53:14.622128 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bcb48f0-3d96-4f77-802d-09079b08782b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:53:14.623169 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bcb48f0-3d96-4f77-802d-09079b08782b None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:14.623832 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bcb48f0-3d96-4f77-802d-09079b08782b None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:14.624308 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bcb48f0-3d96-4f77-802d-09079b08782b None admin] PATH_INFO: `/v3/users/5c4340db782249918182a2e21023cc55` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:14.625857 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5bcb48f0-3d96-4f77-802d-09079b08782b None admin] RBAC: Authorizing `identity:get_user(user_id=5c4340db782249918182a2e21023cc55)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:53:14.627729 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5bcb48f0-3d96-4f77-802d-09079b08782b None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:53:14.629651 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 88/171] 10.222.0.50 () {56 vars in 1177 bytes} [Tue Sep 13 15:53:14 2022] GET /identity/v3/users/5c4340db782249918182a2e21023cc55 => generated 275 bytes in 183 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:53:14.630840 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bcb48f0-3d96-4f77-802d-09079b08782b None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:53:14.693331 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:53:14.813054 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd4e23b2-d6bc-4228-9ab2-dbf75aa2bd71 None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:53:14.826414 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bcb48f0-3d96-4f77-802d-09079b08782b None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:53:14.861189 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:53:14.875676 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-274c4f46-436c-4875-a4a2-f43243598c08 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:53:14.876868 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-274c4f46-436c-4875-a4a2-f43243598c08 None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:14.877798 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-274c4f46-436c-4875-a4a2-f43243598c08 None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:14.878695 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-274c4f46-436c-4875-a4a2-f43243598c08 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:14.891153 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09879f29-d3ab-4420-915a-5dc3e6c100ea None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:53:14.892000 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09879f29-d3ab-4420-915a-5dc3e6c100ea None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:14.892841 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09879f29-d3ab-4420-915a-5dc3e6c100ea None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:14.892841 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09879f29-d3ab-4420-915a-5dc3e6c100ea None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:14.893598 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09879f29-d3ab-4420-915a-5dc3e6c100ea None admin] RBAC: Adding query filter params (role.id=15db4aaca3264ae29bde38903ba43471, scope.project.id=1df61ac8263c4c1e9847d7bdfbcdc627, user.id=5c4340db782249918182a2e21023cc55) {{(pid=81437) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:53:14.893977 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09879f29-d3ab-4420-915a-5dc3e6c100ea None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:53:14.896176 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09879f29-d3ab-4420-915a-5dc3e6c100ea None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:53:14.902937 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 89/172] 10.222.0.50 () {56 vars in 1374 bytes} [Tue Sep 13 15:53:14 2022] GET /identity/v3/role_assignments?user.id=5c4340db782249918182a2e21023cc55&scope.project.id=1df61ac8263c4c1e9847d7bdfbcdc627&role.id=15db4aaca3264ae29bde38903ba43471 => generated 608 bytes in 272 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:53:14.904277 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0d58339-f424-40d9-8a9f-b4d8fc350278 None None] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:14.904651 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0d58339-f424-40d9-8a9f-b4d8fc350278 None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:14.905002 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0d58339-f424-40d9-8a9f-b4d8fc350278 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:14.925667 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.auth.core [None req-f0d58339-f424-40d9-8a9f-b4d8fc350278 None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81437) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:53:14.927203 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-f0d58339-f424-40d9-8a9f-b4d8fc350278 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:53:14.975246 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-274c4f46-436c-4875-a4a2-f43243598c08 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:53:14.977750 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-274c4f46-436c-4875-a4a2-f43243598c08 None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:53:15.008638 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 90/173] 10.222.0.50 () {58 vars in 1024 bytes} [Tue Sep 13 15:53:14 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 13 15:53:15.009617 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0d58339-f424-40d9-8a9f-b4d8fc350278 None None] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:53:15.051783 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:53:15.118071 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 84/174] 10.222.0.50 () {58 vars in 1282 bytes} [Tue Sep 13 15:53:14 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 600 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 13 15:53:15.118770 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-274c4f46-436c-4875-a4a2-f43243598c08 None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:53:15.140055 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0d58339-f424-40d9-8a9f-b4d8fc350278 None None] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:53:15.152182 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:53:15.178962 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-391681a4-454f-4420-8af6-f06c61b51fad None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:53:15.179585 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-391681a4-454f-4420-8af6-f06c61b51fad None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:15.179863 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-391681a4-454f-4420-8af6-f06c61b51fad None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:15.180133 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-391681a4-454f-4420-8af6-f06c61b51fad None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:15.247886 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-274c4f46-436c-4875-a4a2-f43243598c08 None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:53:15.262477 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-391681a4-454f-4420-8af6-f06c61b51fad None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:53:15.264345 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-391681a4-454f-4420-8af6-f06c61b51fad None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:53:15.284828 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5a304c0-4e7a-489e-bb36-ef78626ee2f0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:53:15.285449 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5a304c0-4e7a-489e-bb36-ef78626ee2f0 None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:15.285732 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5a304c0-4e7a-489e-bb36-ef78626ee2f0 None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:15.286012 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5a304c0-4e7a-489e-bb36-ef78626ee2f0 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:15.298813 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:53:15.354097 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5a304c0-4e7a-489e-bb36-ef78626ee2f0 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:53:15.356205 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5a304c0-4e7a-489e-bb36-ef78626ee2f0 None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:53:15.365789 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 91/175] 10.222.0.50 () {58 vars in 1282 bytes} [Tue Sep 13 15:53:15 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 357 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 13 15:53:15.370777 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d468e56c-b08c-4fa8-95a9-99de55b7eadd None None] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:15.370777 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d468e56c-b08c-4fa8-95a9-99de55b7eadd None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:15.370777 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d468e56c-b08c-4fa8-95a9-99de55b7eadd None None] PATH_INFO: `/` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:15.370777 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 92/176] 10.222.0.50 () {54 vars in 927 bytes} [Tue Sep 13 15:53:15 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:53:15.370777 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e0f313a-3377-4009-863a-42fded13f12f None None] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:15.370777 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e0f313a-3377-4009-863a-42fded13f12f None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:15.371276 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e0f313a-3377-4009-863a-42fded13f12f None None] PATH_INFO: `/` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:15.372269 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 93/177] 10.222.0.50 () {54 vars in 927 bytes} [Tue Sep 13 15:53:15 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:53:15.376092 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a67485c-bb0c-45cf-82d4-118431930654 None None] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:15.376412 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a67485c-bb0c-45cf-82d4-118431930654 None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:15.376772 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a67485c-bb0c-45cf-82d4-118431930654 None None] PATH_INFO: `/` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:15.377993 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 94/178] 10.222.0.50 () {54 vars in 927 bytes} [Tue Sep 13 15:53:15 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:53:15.378687 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a67485c-bb0c-45cf-82d4-118431930654 None None] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:53:15.383851 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:53:15.459480 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 85/179] 10.222.0.50 () {58 vars in 1282 bytes} [Tue Sep 13 15:53:15 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 341 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 13 15:53:15.460489 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5a304c0-4e7a-489e-bb36-ef78626ee2f0 None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:53:15.479113 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a67485c-bb0c-45cf-82d4-118431930654 None None] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:53:15.512462 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44335d19-4a9c-4b11-aadf-83c33c052784 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:53:15.513070 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44335d19-4a9c-4b11-aadf-83c33c052784 None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:15.513357 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44335d19-4a9c-4b11-aadf-83c33c052784 None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:15.513632 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44335d19-4a9c-4b11-aadf-83c33c052784 None admin] PATH_INFO: `/v3/projects/423edf83b2b24540924e990bb324af1e` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:15.514748 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44335d19-4a9c-4b11-aadf-83c33c052784 None admin] RBAC: Authorizing `identity:get_project(project_id=423edf83b2b24540924e990bb324af1e)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:53:15.516266 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44335d19-4a9c-4b11-aadf-83c33c052784 None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:53:15.517889 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 95/180] 10.222.0.50 () {56 vars in 1186 bytes} [Tue Sep 13 15:53:15 2022] GET /identity/v3/projects/423edf83b2b24540924e990bb324af1e => generated 298 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:53:15.517889 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44335d19-4a9c-4b11-aadf-83c33c052784 None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:53:15.524842 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:53:15.560252 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5a304c0-4e7a-489e-bb36-ef78626ee2f0 None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:53:15.589356 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:53:15.600940 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-767750e5-6c6c-421b-98d9-143bf1e0da6b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:53:15.601576 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-767750e5-6c6c-421b-98d9-143bf1e0da6b None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:15.601854 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-767750e5-6c6c-421b-98d9-143bf1e0da6b None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:15.602122 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-767750e5-6c6c-421b-98d9-143bf1e0da6b None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:15.612966 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-767750e5-6c6c-421b-98d9-143bf1e0da6b None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 13 15:53:15.617694 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application [None req-767750e5-6c6c-421b-98d9-143bf1e0da6b None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 13 15:53:15.617694 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:53:15.617694 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:53:15.617694 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:53:15.617694 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:53:15.617694 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:53:15.617694 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:53:15.617694 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:53:15.617694 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:53:15.617694 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:53:15.617694 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:53:15.617694 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:53:15.617694 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 13 15:53:15.617694 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 13 15:53:15.617694 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 13 15:53:15.617694 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 13 15:53:15.618581 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 13 15:53:15.618581 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 13 15:53:15.618581 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 13 15:53:15.618581 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 13 15:53:15.618581 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:53:15.618581 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:53:15.618581 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:53:15.618581 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:53:15.618581 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:53:15.618581 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:53:15.618581 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:53:15.618581 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application with Lock( Sep 13 15:53:15.618581 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:53:15.618581 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:53:15.618581 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:53:15.618581 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:53:15.619481 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:53:15.619481 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:53:15.619481 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:53:15.619481 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:53:15.619481 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 13 15:53:15.619481 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 13 15:53:15.619481 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 13 15:53:15.619481 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 13 15:53:15.619481 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 13 15:53:15.619481 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 13 15:53:15.619481 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 13 15:53:15.619481 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Sep 13 15:53:15.619481 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 86/181] 10.222.0.50 () {56 vars in 1096 bytes} [Tue Sep 13 15:53:15 2022] GET /identity/v3/roles/admin => generated 83 bytes in 159 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:53:15.620868 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fd757ee-f591-4d03-b391-b0b8c45596cc None None] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:15.620973 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fd757ee-f591-4d03-b391-b0b8c45596cc None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:15.621292 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fd757ee-f591-4d03-b391-b0b8c45596cc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:15.625375 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44335d19-4a9c-4b11-aadf-83c33c052784 None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:53:15.638835 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.auth.core [None req-9fd757ee-f591-4d03-b391-b0b8c45596cc None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81438) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:53:15.640044 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-9fd757ee-f591-4d03-b391-b0b8c45596cc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:53:15.659783 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39b14fa0-7722-4318-b113-205af862f2b2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:53:15.660414 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39b14fa0-7722-4318-b113-205af862f2b2 None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:15.660691 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39b14fa0-7722-4318-b113-205af862f2b2 None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:15.660965 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39b14fa0-7722-4318-b113-205af862f2b2 None admin] PATH_INFO: `/v3/projects/423edf83b2b24540924e990bb324af1e` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:15.662079 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39b14fa0-7722-4318-b113-205af862f2b2 None admin] RBAC: Authorizing `identity:get_project(project_id=423edf83b2b24540924e990bb324af1e)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:53:15.663576 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39b14fa0-7722-4318-b113-205af862f2b2 None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:53:15.665347 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 96/182] 10.222.0.50 () {56 vars in 1186 bytes} [Tue Sep 13 15:53:15 2022] GET /identity/v3/projects/423edf83b2b24540924e990bb324af1e => generated 298 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:53:15.666439 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99e7aa34-95b9-4873-ae22-a4c8426f33c3 None None] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:15.667310 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99e7aa34-95b9-4873-ae22-a4c8426f33c3 None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:15.667648 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99e7aa34-95b9-4873-ae22-a4c8426f33c3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:15.685940 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.auth.core [None req-99e7aa34-95b9-4873-ae22-a4c8426f33c3 None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81437) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:53:15.687141 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-99e7aa34-95b9-4873-ae22-a4c8426f33c3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:53:15.709876 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 87/183] 10.222.0.50 () {58 vars in 1024 bytes} [Tue Sep 13 15:53:15 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 13 15:53:15.718018 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e21c271-5fe6-4dca-b56f-7eb4a1bcb855 None None] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:15.718018 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e21c271-5fe6-4dca-b56f-7eb4a1bcb855 None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:15.718018 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e21c271-5fe6-4dca-b56f-7eb4a1bcb855 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:15.730236 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.auth.core [None req-2e21c271-5fe6-4dca-b56f-7eb4a1bcb855 None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81438) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:53:15.731451 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-2e21c271-5fe6-4dca-b56f-7eb4a1bcb855 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:53:15.758543 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:53:15.775297 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 97/184] 10.222.0.50 () {58 vars in 1024 bytes} [Tue Sep 13 15:53:15 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 13 15:53:15.775958 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99e7aa34-95b9-4873-ae22-a4c8426f33c3 None None] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:53:15.810934 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 88/185] 10.222.0.50 () {58 vars in 1024 bytes} [Tue Sep 13 15:53:15 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 13 15:53:15.811640 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e21c271-5fe6-4dca-b56f-7eb4a1bcb855 None None] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:53:15.863167 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:53:15.919634 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99e7aa34-95b9-4873-ae22-a4c8426f33c3 None None] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:53:15.972063 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61b5d83f-2978-4bac-8217-771e745ef42b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:53:15.973396 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61b5d83f-2978-4bac-8217-771e745ef42b None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:15.973900 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61b5d83f-2978-4bac-8217-771e745ef42b None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:15.974378 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61b5d83f-2978-4bac-8217-771e745ef42b None admin] PATH_INFO: `/v3/projects/423edf83b2b24540924e990bb324af1e` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:15.975754 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61b5d83f-2978-4bac-8217-771e745ef42b None admin] RBAC: Authorizing `identity:get_project(project_id=423edf83b2b24540924e990bb324af1e)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:53:15.977647 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61b5d83f-2978-4bac-8217-771e745ef42b None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:53:15.980113 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 98/186] 10.222.0.50 () {56 vars in 1186 bytes} [Tue Sep 13 15:53:15 2022] GET /identity/v3/projects/423edf83b2b24540924e990bb324af1e => generated 298 bytes in 205 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:53:15.981091 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61b5d83f-2978-4bac-8217-771e745ef42b None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:53:15.984510 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e21c271-5fe6-4dca-b56f-7eb4a1bcb855 None None] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:53:16.023406 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55b6264a-ae9c-41f2-8cd4-9a3404396e32 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:53:16.024017 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55b6264a-ae9c-41f2-8cd4-9a3404396e32 None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:16.024291 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55b6264a-ae9c-41f2-8cd4-9a3404396e32 None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:16.024561 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55b6264a-ae9c-41f2-8cd4-9a3404396e32 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:16.025279 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55b6264a-ae9c-41f2-8cd4-9a3404396e32 None admin] RBAC: Adding query filter params (role.id=f74e3f6c20bb426ca030f211c3a9aef5, scope.project.id=423edf83b2b24540924e990bb324af1e, user.id=5c4340db782249918182a2e21023cc55) {{(pid=81438) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:53:16.025632 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55b6264a-ae9c-41f2-8cd4-9a3404396e32 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:53:16.027606 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55b6264a-ae9c-41f2-8cd4-9a3404396e32 None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:53:16.034006 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 89/187] 10.222.0.50 () {56 vars in 1374 bytes} [Tue Sep 13 15:53:15 2022] GET /identity/v3/role_assignments?user.id=5c4340db782249918182a2e21023cc55&scope.project.id=423edf83b2b24540924e990bb324af1e&role.id=f74e3f6c20bb426ca030f211c3a9aef5 => generated 608 bytes in 223 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:53:16.035069 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55b6264a-ae9c-41f2-8cd4-9a3404396e32 None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:53:16.084852 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:53:16.145710 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61b5d83f-2978-4bac-8217-771e745ef42b None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:53:16.170755 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55b6264a-ae9c-41f2-8cd4-9a3404396e32 None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:53:16.181031 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:53:16.188256 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-498a673a-9aae-4c3d-8d67-b3f5a3b65437 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:53:16.189137 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-498a673a-9aae-4c3d-8d67-b3f5a3b65437 None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:16.189585 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-498a673a-9aae-4c3d-8d67-b3f5a3b65437 None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:16.190105 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-498a673a-9aae-4c3d-8d67-b3f5a3b65437 None admin] PATH_INFO: `/v3/roles` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:16.191090 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-498a673a-9aae-4c3d-8d67-b3f5a3b65437 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81437) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:53:16.191535 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-498a673a-9aae-4c3d-8d67-b3f5a3b65437 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:53:16.193070 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-498a673a-9aae-4c3d-8d67-b3f5a3b65437 None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:53:16.203329 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 99/188] 10.222.0.50 () {56 vars in 1099 bytes} [Tue Sep 13 15:53:15 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 222 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:53:16.204625 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-498a673a-9aae-4c3d-8d67-b3f5a3b65437 None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:53:16.217907 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-163fe6a2-fce8-41db-a765-a51772c2c125 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:53:16.218607 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-163fe6a2-fce8-41db-a765-a51772c2c125 None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:16.219237 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-163fe6a2-fce8-41db-a765-a51772c2c125 None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:16.219511 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-163fe6a2-fce8-41db-a765-a51772c2c125 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:16.220252 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-163fe6a2-fce8-41db-a765-a51772c2c125 None admin] RBAC: Adding query filter params (role.id=031d4e1a23e34f71858fc4a6dfbd4ccb, scope.project.id=423edf83b2b24540924e990bb324af1e, user.id=e0a915a0eb3c42c7af97dcc7eed5b42f) {{(pid=81438) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:53:16.220561 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-163fe6a2-fce8-41db-a765-a51772c2c125 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:53:16.222010 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-163fe6a2-fce8-41db-a765-a51772c2c125 None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:53:16.228985 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 90/189] 10.222.0.50 () {56 vars in 1374 bytes} [Tue Sep 13 15:53:16 2022] GET /identity/v3/role_assignments?user.id=e0a915a0eb3c42c7af97dcc7eed5b42f&scope.project.id=423edf83b2b24540924e990bb324af1e&role.id=031d4e1a23e34f71858fc4a6dfbd4ccb => generated 608 bytes in 194 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:53:16.235999 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-163fe6a2-fce8-41db-a765-a51772c2c125 None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:53:16.351634 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-498a673a-9aae-4c3d-8d67-b3f5a3b65437 None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:53:16.379690 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-163fe6a2-fce8-41db-a765-a51772c2c125 None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:53:16.388308 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:53:16.395711 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4f437d6-7da4-4221-bf19-dbb24f5836a3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:53:16.396574 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4f437d6-7da4-4221-bf19-dbb24f5836a3 None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:16.396918 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4f437d6-7da4-4221-bf19-dbb24f5836a3 None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:16.397634 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4f437d6-7da4-4221-bf19-dbb24f5836a3 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:16.398820 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4f437d6-7da4-4221-bf19-dbb24f5836a3 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:53:16.400337 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4f437d6-7da4-4221-bf19-dbb24f5836a3 None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:53:16.406422 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 100/190] 10.222.0.50 () {56 vars in 1108 bytes} [Tue Sep 13 15:53:16 2022] GET /identity/v3/domains/default => generated 199 bytes in 198 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:53:16.406422 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4f437d6-7da4-4221-bf19-dbb24f5836a3 None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:53:16.425430 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95491b10-3f8f-4e72-863b-a12267949cd4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:53:16.426312 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95491b10-3f8f-4e72-863b-a12267949cd4 None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:16.426596 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95491b10-3f8f-4e72-863b-a12267949cd4 None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:16.426869 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95491b10-3f8f-4e72-863b-a12267949cd4 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:16.428140 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95491b10-3f8f-4e72-863b-a12267949cd4 None admin] RBAC: Adding query filter params (role.id=15db4aaca3264ae29bde38903ba43471, scope.project.id=423edf83b2b24540924e990bb324af1e, user.id=5c4340db782249918182a2e21023cc55) {{(pid=81438) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:53:16.428140 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95491b10-3f8f-4e72-863b-a12267949cd4 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:53:16.429356 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95491b10-3f8f-4e72-863b-a12267949cd4 None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:53:16.438413 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 91/191] 10.222.0.50 () {56 vars in 1374 bytes} [Tue Sep 13 15:53:16 2022] GET /identity/v3/role_assignments?user.id=5c4340db782249918182a2e21023cc55&scope.project.id=423edf83b2b24540924e990bb324af1e&role.id=15db4aaca3264ae29bde38903ba43471 => generated 608 bytes in 209 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:53:16.439376 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95491b10-3f8f-4e72-863b-a12267949cd4 None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:53:16.486200 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:53:16.543779 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4f437d6-7da4-4221-bf19-dbb24f5836a3 None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:53:16.579396 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95491b10-3f8f-4e72-863b-a12267949cd4 None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:53:16.589140 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54ef591b-8864-4468-b475-116aa31eae09 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:53:16.589892 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54ef591b-8864-4468-b475-116aa31eae09 None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:16.590174 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54ef591b-8864-4468-b475-116aa31eae09 None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:16.590713 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54ef591b-8864-4468-b475-116aa31eae09 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:16.602727 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54ef591b-8864-4468-b475-116aa31eae09 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 13 15:53:16.609011 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application [None req-54ef591b-8864-4468-b475-116aa31eae09 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Sep 13 15:53:16.609011 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:53:16.609011 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:53:16.609011 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:53:16.609011 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:53:16.609011 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:53:16.609011 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:53:16.609011 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:53:16.609011 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:53:16.609011 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:53:16.609011 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:53:16.609011 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:53:16.609011 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 13 15:53:16.609011 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 13 15:53:16.609011 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 13 15:53:16.609011 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 13 15:53:16.609956 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 13 15:53:16.609956 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 13 15:53:16.609956 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 13 15:53:16.609956 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 13 15:53:16.609956 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:53:16.609956 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:53:16.609956 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:53:16.609956 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:53:16.609956 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:53:16.609956 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:53:16.609956 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:53:16.609956 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application with Lock( Sep 13 15:53:16.609956 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:53:16.609956 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:53:16.609956 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:53:16.609956 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:53:16.611549 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:53:16.611549 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:53:16.611549 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:53:16.611549 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:53:16.611549 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 13 15:53:16.611549 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 13 15:53:16.611549 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 13 15:53:16.611549 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 13 15:53:16.611549 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 13 15:53:16.611549 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 13 15:53:16.611549 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Sep 13 15:53:16.611549 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application Sep 13 15:53:16.611549 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 101/192] 10.222.0.50 () {56 vars in 1099 bytes} [Tue Sep 13 15:53:16 2022] GET /identity/v3/roles/reader => generated 84 bytes in 208 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:53:16.611549 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54ef591b-8864-4468-b475-116aa31eae09 None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:53:16.621187 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e58fd7b-2647-4b9f-b971-bc2c5ee0b115 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:53:16.621844 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e58fd7b-2647-4b9f-b971-bc2c5ee0b115 None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:16.622127 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e58fd7b-2647-4b9f-b971-bc2c5ee0b115 None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:16.622409 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e58fd7b-2647-4b9f-b971-bc2c5ee0b115 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:16.633772 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e58fd7b-2647-4b9f-b971-bc2c5ee0b115 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 13 15:53:16.639908 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application [None req-5e58fd7b-2647-4b9f-b971-bc2c5ee0b115 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 13 15:53:16.639908 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:53:16.639908 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:53:16.639908 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:53:16.639908 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:53:16.639908 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:53:16.639908 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:53:16.639908 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:53:16.639908 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:53:16.639908 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:53:16.639908 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:53:16.639908 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:53:16.639908 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 13 15:53:16.639908 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 13 15:53:16.639908 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 13 15:53:16.639908 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 13 15:53:16.640999 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 13 15:53:16.640999 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 13 15:53:16.640999 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 13 15:53:16.640999 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 13 15:53:16.640999 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:53:16.640999 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:53:16.640999 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:53:16.640999 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:53:16.640999 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:53:16.640999 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:53:16.640999 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:53:16.640999 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application with Lock( Sep 13 15:53:16.640999 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:53:16.640999 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:53:16.640999 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:53:16.640999 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:53:16.641963 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:53:16.641963 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:53:16.641963 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:53:16.641963 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:53:16.641963 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 13 15:53:16.641963 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 13 15:53:16.641963 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 13 15:53:16.641963 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 13 15:53:16.641963 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 13 15:53:16.641963 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 13 15:53:16.641963 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 13 15:53:16.641963 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Sep 13 15:53:16.641963 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 92/193] 10.222.0.50 () {56 vars in 1114 bytes} [Tue Sep 13 15:53:16 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 202 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:53:16.641963 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e58fd7b-2647-4b9f-b971-bc2c5ee0b115 None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:53:16.688114 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:53:16.735459 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54ef591b-8864-4468-b475-116aa31eae09 None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:53:16.749937 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:53:16.755976 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e58fd7b-2647-4b9f-b971-bc2c5ee0b115 None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:53:16.770945 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11b97d0c-1e58-441f-9f14-fa180fa0a1dc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:53:16.771781 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11b97d0c-1e58-441f-9f14-fa180fa0a1dc None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:16.772065 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11b97d0c-1e58-441f-9f14-fa180fa0a1dc None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:16.772345 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11b97d0c-1e58-441f-9f14-fa180fa0a1dc None admin] PATH_INFO: `/v3/users/ff3f9868e06e488a99e9902368f1a78a` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:16.785741 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11b97d0c-1e58-441f-9f14-fa180fa0a1dc None admin] RBAC: Authorizing `identity:get_user(user_id=ff3f9868e06e488a99e9902368f1a78a)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:53:16.787251 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11b97d0c-1e58-441f-9f14-fa180fa0a1dc None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:53:16.788870 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 102/194] 10.222.0.50 () {56 vars in 1177 bytes} [Tue Sep 13 15:53:16 2022] GET /identity/v3/users/ff3f9868e06e488a99e9902368f1a78a => generated 283 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:53:16.790021 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-191c2f20-09a5-417f-b3e1-79d6c69777b3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:53:16.790634 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-191c2f20-09a5-417f-b3e1-79d6c69777b3 None admin] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:16.790733 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11b97d0c-1e58-441f-9f14-fa180fa0a1dc None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:53:16.791059 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-191c2f20-09a5-417f-b3e1-79d6c69777b3 None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:16.791266 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-191c2f20-09a5-417f-b3e1-79d6c69777b3 None admin] PATH_INFO: `/v3/users` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:16.791950 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-191c2f20-09a5-417f-b3e1-79d6c69777b3 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:53:16.793599 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-191c2f20-09a5-417f-b3e1-79d6c69777b3 None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:53:16.814265 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 93/195] 10.222.0.50 () {60 vars in 1132 bytes} [Tue Sep 13 15:53:16 2022] POST /identity/v3/users => generated 297 bytes in 173 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 15:53:16.815200 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-191c2f20-09a5-417f-b3e1-79d6c69777b3 None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:53:16.898997 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11b97d0c-1e58-441f-9f14-fa180fa0a1dc None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:53:16.916456 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:53:16.935405 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3440affe-50db-4ef7-b065-7c7abb38d7ad None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:53:16.936030 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3440affe-50db-4ef7-b065-7c7abb38d7ad None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:16.936308 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3440affe-50db-4ef7-b065-7c7abb38d7ad None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:16.936578 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3440affe-50db-4ef7-b065-7c7abb38d7ad None admin] PATH_INFO: `/v3/roles` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:16.937253 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3440affe-50db-4ef7-b065-7c7abb38d7ad None admin] RBAC: Adding query filter params (name=reader) {{(pid=81437) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:53:16.937548 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3440affe-50db-4ef7-b065-7c7abb38d7ad None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:53:16.938994 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3440affe-50db-4ef7-b065-7c7abb38d7ad None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:53:16.940822 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-191c2f20-09a5-417f-b3e1-79d6c69777b3 None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:53:16.950949 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 103/196] 10.222.0.50 () {56 vars in 1101 bytes} [Tue Sep 13 15:53:16 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:53:16.950949 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3440affe-50db-4ef7-b065-7c7abb38d7ad None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:53:16.982970 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c261bf91-9ada-43d6-9341-1ac83c157bb4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:53:16.983589 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c261bf91-9ada-43d6-9341-1ac83c157bb4 None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:16.983862 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c261bf91-9ada-43d6-9341-1ac83c157bb4 None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:16.984130 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c261bf91-9ada-43d6-9341-1ac83c157bb4 None admin] PATH_INFO: `/v3/roles` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:16.984799 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c261bf91-9ada-43d6-9341-1ac83c157bb4 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81438) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:53:16.985109 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c261bf91-9ada-43d6-9341-1ac83c157bb4 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:53:16.986472 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c261bf91-9ada-43d6-9341-1ac83c157bb4 None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:53:17.000365 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 94/197] 10.222.0.50 () {56 vars in 1111 bytes} [Tue Sep 13 15:53:16 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:53:17.000365 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c261bf91-9ada-43d6-9341-1ac83c157bb4 None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:53:17.018690 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:53:17.087862 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3440affe-50db-4ef7-b065-7c7abb38d7ad None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:53:17.131092 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c261bf91-9ada-43d6-9341-1ac83c157bb4 None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:53:17.134312 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-994fadda-818e-4c20-9c5e-21864b473f17 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:53:17.137183 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-994fadda-818e-4c20-9c5e-21864b473f17 None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:17.137183 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-994fadda-818e-4c20-9c5e-21864b473f17 None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:17.137183 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-994fadda-818e-4c20-9c5e-21864b473f17 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:17.167653 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b4df3bc-6f2d-49ca-8fd3-ee11b13d4dae None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:53:17.168356 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b4df3bc-6f2d-49ca-8fd3-ee11b13d4dae None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:17.168639 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b4df3bc-6f2d-49ca-8fd3-ee11b13d4dae None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:17.168912 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b4df3bc-6f2d-49ca-8fd3-ee11b13d4dae None admin] PATH_INFO: `/v3/users/83ce764787ea4cc389a55bbb8d35c45f` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:17.170158 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b4df3bc-6f2d-49ca-8fd3-ee11b13d4dae None admin] RBAC: Authorizing `identity:get_user(user_id=83ce764787ea4cc389a55bbb8d35c45f)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:53:17.171745 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b4df3bc-6f2d-49ca-8fd3-ee11b13d4dae None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:53:17.174482 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 95/198] 10.222.0.50 () {56 vars in 1177 bytes} [Tue Sep 13 15:53:16 2022] GET /identity/v3/users/83ce764787ea4cc389a55bbb8d35c45f => generated 289 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:53:17.175321 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b4df3bc-6f2d-49ca-8fd3-ee11b13d4dae None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:53:17.194016 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:53:17.226071 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-994fadda-818e-4c20-9c5e-21864b473f17 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:53:17.227664 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-994fadda-818e-4c20-9c5e-21864b473f17 None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:53:17.293632 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:53:17.318176 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b4df3bc-6f2d-49ca-8fd3-ee11b13d4dae None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:53:17.342108 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 104/199] 10.222.0.50 () {58 vars in 1282 bytes} [Tue Sep 13 15:53:16 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 393 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 13 15:53:17.347092 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-994fadda-818e-4c20-9c5e-21864b473f17 None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:53:17.363496 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f9d6447-c2a6-43f4-9021-20fb804a0402 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:53:17.364140 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f9d6447-c2a6-43f4-9021-20fb804a0402 None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:17.364420 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f9d6447-c2a6-43f4-9021-20fb804a0402 None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:17.365505 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f9d6447-c2a6-43f4-9021-20fb804a0402 None admin] PATH_INFO: `/v3/users/ff3f9868e06e488a99e9902368f1a78a` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:17.367777 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f9d6447-c2a6-43f4-9021-20fb804a0402 None admin] RBAC: Authorizing `identity:get_user(user_id=ff3f9868e06e488a99e9902368f1a78a)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:53:17.369813 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f9d6447-c2a6-43f4-9021-20fb804a0402 None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:53:17.372642 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 96/200] 10.222.0.50 () {56 vars in 1177 bytes} [Tue Sep 13 15:53:17 2022] GET /identity/v3/users/ff3f9868e06e488a99e9902368f1a78a => generated 283 bytes in 197 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:53:17.373727 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f9d6447-c2a6-43f4-9021-20fb804a0402 None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:53:17.447711 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:53:17.466811 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-994fadda-818e-4c20-9c5e-21864b473f17 None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:53:17.505283 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28ebc3b3-ea92-4aec-84fd-2901005e5101 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:53:17.506016 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28ebc3b3-ea92-4aec-84fd-2901005e5101 None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:17.506297 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28ebc3b3-ea92-4aec-84fd-2901005e5101 None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:17.506763 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28ebc3b3-ea92-4aec-84fd-2901005e5101 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:17.524911 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f9d6447-c2a6-43f4-9021-20fb804a0402 None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:53:17.568589 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a9b7c63-7abf-4baf-8b95-e33bdbfec3e0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:53:17.569328 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a9b7c63-7abf-4baf-8b95-e33bdbfec3e0 None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:17.569943 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a9b7c63-7abf-4baf-8b95-e33bdbfec3e0 None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:17.570302 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a9b7c63-7abf-4baf-8b95-e33bdbfec3e0 None admin] PATH_INFO: `/v3/projects/9ec1adca55204df2b653531664b60123` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:17.571910 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a9b7c63-7abf-4baf-8b95-e33bdbfec3e0 None admin] RBAC: Authorizing `identity:get_project(project_id=9ec1adca55204df2b653531664b60123)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:53:17.574301 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a9b7c63-7abf-4baf-8b95-e33bdbfec3e0 None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:53:17.576411 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 97/201] 10.222.0.50 () {56 vars in 1186 bytes} [Tue Sep 13 15:53:17 2022] GET /identity/v3/projects/9ec1adca55204df2b653531664b60123 => generated 302 bytes in 203 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:53:17.577378 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a9b7c63-7abf-4baf-8b95-e33bdbfec3e0 None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:53:17.600438 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:53:17.608203 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28ebc3b3-ea92-4aec-84fd-2901005e5101 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:53:17.610835 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28ebc3b3-ea92-4aec-84fd-2901005e5101 None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:53:17.701967 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a9b7c63-7abf-4baf-8b95-e33bdbfec3e0 None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:53:17.711612 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:53:17.743357 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbd8ca65-55d1-4cb5-8cd5-e2231cd1f857 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:53:17.743861 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 105/202] 10.222.0.50 () {58 vars in 1282 bytes} [Tue Sep 13 15:53:17 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 401 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 13 15:53:17.744002 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbd8ca65-55d1-4cb5-8cd5-e2231cd1f857 None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:17.744149 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbd8ca65-55d1-4cb5-8cd5-e2231cd1f857 None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:17.744534 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbd8ca65-55d1-4cb5-8cd5-e2231cd1f857 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:17.746087 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28ebc3b3-ea92-4aec-84fd-2901005e5101 None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:53:17.828443 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbd8ca65-55d1-4cb5-8cd5-e2231cd1f857 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:53:17.830004 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbd8ca65-55d1-4cb5-8cd5-e2231cd1f857 None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:53:17.851722 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:53:17.874651 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28ebc3b3-ea92-4aec-84fd-2901005e5101 None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:53:17.920622 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eea19204-6b3d-4e4d-9e5a-75d5592b3250 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:53:17.921644 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eea19204-6b3d-4e4d-9e5a-75d5592b3250 None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:17.922153 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eea19204-6b3d-4e4d-9e5a-75d5592b3250 None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:17.922658 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eea19204-6b3d-4e4d-9e5a-75d5592b3250 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:17.923688 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eea19204-6b3d-4e4d-9e5a-75d5592b3250 None admin] RBAC: Adding query filter params (role.id=031d4e1a23e34f71858fc4a6dfbd4ccb, scope.project.id=9ec1adca55204df2b653531664b60123, user.id=ff3f9868e06e488a99e9902368f1a78a) {{(pid=81437) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:53:17.924206 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eea19204-6b3d-4e4d-9e5a-75d5592b3250 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:53:17.925873 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eea19204-6b3d-4e4d-9e5a-75d5592b3250 None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:53:17.932963 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 106/203] 10.222.0.50 () {56 vars in 1374 bytes} [Tue Sep 13 15:53:17 2022] GET /identity/v3/role_assignments?user.id=ff3f9868e06e488a99e9902368f1a78a&scope.project.id=9ec1adca55204df2b653531664b60123&role.id=031d4e1a23e34f71858fc4a6dfbd4ccb => generated 259 bytes in 187 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:53:17.934069 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eea19204-6b3d-4e4d-9e5a-75d5592b3250 None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:53:17.957613 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 98/204] 10.222.0.50 () {58 vars in 1282 bytes} [Tue Sep 13 15:53:17 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 380 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 13 15:53:17.963003 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbd8ca65-55d1-4cb5-8cd5-e2231cd1f857 None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:53:18.011050 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:53:18.108772 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eea19204-6b3d-4e4d-9e5a-75d5592b3250 None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:53:18.151419 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e350f87-95c8-47a7-a303-4068ded0196e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:53:18.153259 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e350f87-95c8-47a7-a303-4068ded0196e None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:18.153715 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e350f87-95c8-47a7-a303-4068ded0196e None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:18.154147 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e350f87-95c8-47a7-a303-4068ded0196e None admin] PATH_INFO: `/v3/projects/423edf83b2b24540924e990bb324af1e` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:18.155497 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e350f87-95c8-47a7-a303-4068ded0196e None admin] RBAC: Authorizing `identity:get_project(project_id=423edf83b2b24540924e990bb324af1e)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:53:18.157104 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e350f87-95c8-47a7-a303-4068ded0196e None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:53:18.159524 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 107/205] 10.222.0.50 () {56 vars in 1186 bytes} [Tue Sep 13 15:53:17 2022] GET /identity/v3/projects/423edf83b2b24540924e990bb324af1e => generated 298 bytes in 226 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:53:18.165076 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e350f87-95c8-47a7-a303-4068ded0196e None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:53:18.210779 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbd8ca65-55d1-4cb5-8cd5-e2231cd1f857 None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:53:18.244599 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:53:18.279279 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31765a76-a4fb-45f2-b6c1-8dd85ce14f18 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:53:18.279864 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31765a76-a4fb-45f2-b6c1-8dd85ce14f18 None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:18.280148 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31765a76-a4fb-45f2-b6c1-8dd85ce14f18 None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:18.280424 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31765a76-a4fb-45f2-b6c1-8dd85ce14f18 None admin] PATH_INFO: `/v3/projects/9ec1adca55204df2b653531664b60123` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:18.281695 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31765a76-a4fb-45f2-b6c1-8dd85ce14f18 None admin] RBAC: Authorizing `identity:get_project(project_id=9ec1adca55204df2b653531664b60123)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:53:18.288226 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e350f87-95c8-47a7-a303-4068ded0196e None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:53:18.291377 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31765a76-a4fb-45f2-b6c1-8dd85ce14f18 None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:53:18.294300 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 99/206] 10.222.0.50 () {56 vars in 1186 bytes} [Tue Sep 13 15:53:17 2022] GET /identity/v3/projects/9ec1adca55204df2b653531664b60123 => generated 302 bytes in 332 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 13 15:53:18.300785 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31765a76-a4fb-45f2-b6c1-8dd85ce14f18 None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:53:18.307900 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:53:18.356581 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01f291ff-fd8c-4aa4-af5c-9bf4f0a77c10 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:53:18.357204 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01f291ff-fd8c-4aa4-af5c-9bf4f0a77c10 None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:18.357483 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01f291ff-fd8c-4aa4-af5c-9bf4f0a77c10 None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:18.357756 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01f291ff-fd8c-4aa4-af5c-9bf4f0a77c10 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:18.358493 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01f291ff-fd8c-4aa4-af5c-9bf4f0a77c10 None admin] RBAC: Adding query filter params (role.id=b2d1670446a54085ad2bdb935f003d25, scope.project.id=423edf83b2b24540924e990bb324af1e, user.id=83ce764787ea4cc389a55bbb8d35c45f) {{(pid=81437) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:53:18.358802 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01f291ff-fd8c-4aa4-af5c-9bf4f0a77c10 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:53:18.360284 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01f291ff-fd8c-4aa4-af5c-9bf4f0a77c10 None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:53:18.374696 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 108/207] 10.222.0.50 () {56 vars in 1374 bytes} [Tue Sep 13 15:53:18 2022] GET /identity/v3/role_assignments?user.id=83ce764787ea4cc389a55bbb8d35c45f&scope.project.id=423edf83b2b24540924e990bb324af1e&role.id=b2d1670446a54085ad2bdb935f003d25 => generated 608 bytes in 210 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 13 15:53:18.417433 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31765a76-a4fb-45f2-b6c1-8dd85ce14f18 None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:53:18.457579 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-096bbc3a-e094-46d4-a510-544b8939ad74 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:53:18.458192 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-096bbc3a-e094-46d4-a510-544b8939ad74 None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:18.458474 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-096bbc3a-e094-46d4-a510-544b8939ad74 None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:18.458748 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-096bbc3a-e094-46d4-a510-544b8939ad74 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:18.459514 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-096bbc3a-e094-46d4-a510-544b8939ad74 None admin] RBAC: Adding query filter params (role.id=f74e3f6c20bb426ca030f211c3a9aef5, scope.project.id=9ec1adca55204df2b653531664b60123, user.id=ff3f9868e06e488a99e9902368f1a78a) {{(pid=81438) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:53:18.459825 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-096bbc3a-e094-46d4-a510-544b8939ad74 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:53:18.461323 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-096bbc3a-e094-46d4-a510-544b8939ad74 None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:53:18.467581 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 100/208] 10.222.0.50 () {56 vars in 1374 bytes} [Tue Sep 13 15:53:18 2022] GET /identity/v3/role_assignments?user.id=ff3f9868e06e488a99e9902368f1a78a&scope.project.id=9ec1adca55204df2b653531664b60123&role.id=f74e3f6c20bb426ca030f211c3a9aef5 => generated 259 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 13 15:53:19.355351 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d917c36-a584-4d50-8e30-621fc3f5ec5a None None] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:19.355896 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d917c36-a584-4d50-8e30-621fc3f5ec5a None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:19.355896 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d917c36-a584-4d50-8e30-621fc3f5ec5a None None] PATH_INFO: `/` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:19.357233 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 109/209] 10.222.0.50 () {54 vars in 927 bytes} [Tue Sep 13 15:53:19 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:53:19.362805 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aef1a45f-9b87-4cd6-a0d1-a1ef7b435cfa None None] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:19.363242 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aef1a45f-9b87-4cd6-a0d1-a1ef7b435cfa None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:19.363432 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aef1a45f-9b87-4cd6-a0d1-a1ef7b435cfa None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:19.382495 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.auth.core [None req-aef1a45f-9b87-4cd6-a0d1-a1ef7b435cfa None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81438) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:53:19.383781 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-aef1a45f-9b87-4cd6-a0d1-a1ef7b435cfa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:53:19.452260 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:53:19.484973 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 101/210] 10.222.0.50 () {58 vars in 1024 bytes} [Tue Sep 13 15:53:19 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 123 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 13 15:53:19.538889 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5c6a557-8e32-4f96-8011-d53bfcfc97f2 None None] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:19.539239 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5c6a557-8e32-4f96-8011-d53bfcfc97f2 None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:19.539468 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5c6a557-8e32-4f96-8011-d53bfcfc97f2 None None] PATH_INFO: `/` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:19.540958 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 110/211] 10.222.0.50 () {54 vars in 927 bytes} [Tue Sep 13 15:53:19 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:53:19.560698 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcfe7ece-6a23-49b1-8a41-7f5acc4d5a27 None None] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:19.560698 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcfe7ece-6a23-49b1-8a41-7f5acc4d5a27 None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:19.560698 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcfe7ece-6a23-49b1-8a41-7f5acc4d5a27 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:19.579193 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.auth.core [None req-fcfe7ece-6a23-49b1-8a41-7f5acc4d5a27 None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81438) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:53:19.580371 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-fcfe7ece-6a23-49b1-8a41-7f5acc4d5a27 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:53:19.673609 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 102/212] 10.222.0.50 () {58 vars in 1024 bytes} [Tue Sep 13 15:53:19 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 118 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 13 15:53:19.803347 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5c6a557-8e32-4f96-8011-d53bfcfc97f2 None None] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:53:19.908780 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5c6a557-8e32-4f96-8011-d53bfcfc97f2 None None] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:53:19.945297 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3a1b5e3-6a15-4e62-941c-ae62e45f1c59 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:53:19.945911 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3a1b5e3-6a15-4e62-941c-ae62e45f1c59 None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:19.946195 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3a1b5e3-6a15-4e62-941c-ae62e45f1c59 None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:19.946467 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3a1b5e3-6a15-4e62-941c-ae62e45f1c59 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:19.947599 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3a1b5e3-6a15-4e62-941c-ae62e45f1c59 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:53:19.949079 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3a1b5e3-6a15-4e62-941c-ae62e45f1c59 None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:53:19.951001 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 111/213] 10.222.0.50 () {56 vars in 1108 bytes} [Tue Sep 13 15:53:19 2022] GET /identity/v3/domains/default => generated 199 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:53:19.956350 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcfe7ece-6a23-49b1-8a41-7f5acc4d5a27 None None] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:53:19.987129 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3a1b5e3-6a15-4e62-941c-ae62e45f1c59 None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:53:20.045343 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:53:20.091694 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcfe7ece-6a23-49b1-8a41-7f5acc4d5a27 None None] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:53:20.106959 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:53:20.117784 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3a1b5e3-6a15-4e62-941c-ae62e45f1c59 None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:53:20.140718 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37732b1c-03d3-4162-b09a-2cb9fc706dfa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:53:20.141356 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37732b1c-03d3-4162-b09a-2cb9fc706dfa None admin] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:20.141637 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37732b1c-03d3-4162-b09a-2cb9fc706dfa None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:20.141908 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37732b1c-03d3-4162-b09a-2cb9fc706dfa None admin] PATH_INFO: `/v3/users` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:20.142627 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37732b1c-03d3-4162-b09a-2cb9fc706dfa None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:53:20.144160 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37732b1c-03d3-4162-b09a-2cb9fc706dfa None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:53:20.157805 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66067c29-d8ad-4ca9-ad51-3a085c0ed248 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:53:20.158937 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66067c29-d8ad-4ca9-ad51-3a085c0ed248 None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:20.158937 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66067c29-d8ad-4ca9-ad51-3a085c0ed248 None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:20.159831 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66067c29-d8ad-4ca9-ad51-3a085c0ed248 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:20.175063 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66067c29-d8ad-4ca9-ad51-3a085c0ed248 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 13 15:53:20.175481 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 103/214] 10.222.0.50 () {60 vars in 1132 bytes} [Tue Sep 13 15:53:19 2022] POST /identity/v3/users => generated 297 bytes in 213 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 15:53:20.177481 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application [None req-66067c29-d8ad-4ca9-ad51-3a085c0ed248 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 13 15:53:20.177481 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:53:20.177481 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:53:20.177481 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:53:20.177481 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:53:20.177481 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:53:20.177481 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:53:20.177481 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:53:20.177481 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:53:20.177481 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:53:20.177481 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:53:20.177481 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:53:20.177481 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 13 15:53:20.177481 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 13 15:53:20.177481 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 13 15:53:20.177481 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 13 15:53:20.178320 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 13 15:53:20.178320 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 13 15:53:20.178320 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 13 15:53:20.178320 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 13 15:53:20.178320 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:53:20.178320 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:53:20.178320 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:53:20.178320 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:53:20.178320 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:53:20.178320 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:53:20.178320 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:53:20.178320 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application with Lock( Sep 13 15:53:20.178320 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:53:20.178320 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:53:20.178320 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:53:20.178320 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:53:20.179481 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:53:20.179481 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:53:20.179481 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:53:20.179481 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:53:20.179481 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 13 15:53:20.179481 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 13 15:53:20.179481 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 13 15:53:20.179481 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 13 15:53:20.179481 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 13 15:53:20.179481 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 13 15:53:20.179481 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 13 15:53:20.179481 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application Sep 13 15:53:20.183456 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 112/215] 10.222.0.50 () {56 vars in 1099 bytes} [Tue Sep 13 15:53:19 2022] GET /identity/v3/roles/member => generated 84 bytes in 193 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 13 15:53:20.185504 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37732b1c-03d3-4162-b09a-2cb9fc706dfa None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:53:20.297885 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37732b1c-03d3-4162-b09a-2cb9fc706dfa None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:53:20.344344 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6141e892-c642-494c-9c40-55344c090fe6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:53:20.345419 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6141e892-c642-494c-9c40-55344c090fe6 None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:20.345900 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6141e892-c642-494c-9c40-55344c090fe6 None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:20.346394 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6141e892-c642-494c-9c40-55344c090fe6 None admin] PATH_INFO: `/v3/roles` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:20.347382 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6141e892-c642-494c-9c40-55344c090fe6 None admin] RBAC: Adding query filter params (name=member) {{(pid=81438) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:53:20.347962 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6141e892-c642-494c-9c40-55344c090fe6 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:53:20.349774 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6141e892-c642-494c-9c40-55344c090fe6 None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:53:20.360558 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 104/216] 10.222.0.50 () {56 vars in 1101 bytes} [Tue Sep 13 15:53:20 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:53:20.370991 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66067c29-d8ad-4ca9-ad51-3a085c0ed248 None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:53:20.445259 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-471f6f4f-f22e-4121-b5e0-001f89c19646 None None] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:20.445760 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-471f6f4f-f22e-4121-b5e0-001f89c19646 None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:20.445836 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-471f6f4f-f22e-4121-b5e0-001f89c19646 None None] PATH_INFO: `/` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:20.447181 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 105/217] 10.222.0.50 () {54 vars in 927 bytes} [Tue Sep 13 15:53:20 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 13 15:53:20.453213 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d20ddfe5-0dfa-4cef-961d-df31eb246591 None None] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:20.453496 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d20ddfe5-0dfa-4cef-961d-df31eb246591 None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:20.453771 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d20ddfe5-0dfa-4cef-961d-df31eb246591 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:20.474624 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.auth.core [None req-d20ddfe5-0dfa-4cef-961d-df31eb246591 None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81438) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:53:20.475855 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-d20ddfe5-0dfa-4cef-961d-df31eb246591 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:53:20.482584 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66067c29-d8ad-4ca9-ad51-3a085c0ed248 None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:53:20.485295 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:53:20.507916 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:53:20.532063 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2bb2ae0-2633-48a5-bc17-c199266c4bf3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:53:20.532698 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2bb2ae0-2633-48a5-bc17-c199266c4bf3 None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:20.532977 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2bb2ae0-2633-48a5-bc17-c199266c4bf3 None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:20.533246 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2bb2ae0-2633-48a5-bc17-c199266c4bf3 None admin] PATH_INFO: `/v3/users/b8269ec40ba14c51825ec2be9c71b254` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:20.552095 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2bb2ae0-2633-48a5-bc17-c199266c4bf3 None admin] RBAC: Authorizing `identity:get_user(user_id=b8269ec40ba14c51825ec2be9c71b254)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:53:20.553700 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2bb2ae0-2633-48a5-bc17-c199266c4bf3 None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:53:20.556167 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 113/218] 10.222.0.50 () {56 vars in 1177 bytes} [Tue Sep 13 15:53:20 2022] GET /identity/v3/users/b8269ec40ba14c51825ec2be9c71b254 => generated 297 bytes in 190 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:53:20.563577 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2bb2ae0-2633-48a5-bc17-c199266c4bf3 None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:53:20.579364 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 106/219] 10.222.0.50 () {58 vars in 1024 bytes} [Tue Sep 13 15:53:20 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 127 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 13 15:53:20.676128 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2bb2ae0-2633-48a5-bc17-c199266c4bf3 None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:53:20.717192 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dda4dfab-b74c-401f-91f3-f3a9796642e0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:53:20.717809 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dda4dfab-b74c-401f-91f3-f3a9796642e0 None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:20.718140 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dda4dfab-b74c-401f-91f3-f3a9796642e0 None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:20.718416 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dda4dfab-b74c-401f-91f3-f3a9796642e0 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:20.749650 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:53:20.810574 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dda4dfab-b74c-401f-91f3-f3a9796642e0 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:53:20.812000 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dda4dfab-b74c-401f-91f3-f3a9796642e0 None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:53:20.891197 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d20ddfe5-0dfa-4cef-961d-df31eb246591 None None] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:53:20.953848 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 114/220] 10.222.0.50 () {58 vars in 1282 bytes} [Tue Sep 13 15:53:20 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 391 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 13 15:53:20.955432 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb8d948c-cc43-4ff1-9dd1-70104db4a0cf None None] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:20.955926 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb8d948c-cc43-4ff1-9dd1-70104db4a0cf None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:20.956379 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb8d948c-cc43-4ff1-9dd1-70104db4a0cf None None] PATH_INFO: `/` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:20.957913 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 115/221] 10.222.0.50 () {54 vars in 927 bytes} [Tue Sep 13 15:53:20 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:53:20.958987 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb8d948c-cc43-4ff1-9dd1-70104db4a0cf None None] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:53:21.014929 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d20ddfe5-0dfa-4cef-961d-df31eb246591 None None] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:53:21.031705 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:53:21.057802 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe2c5c42-7b1c-462c-9a6a-7bc7bec58acb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:53:21.058537 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe2c5c42-7b1c-462c-9a6a-7bc7bec58acb None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:21.058927 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe2c5c42-7b1c-462c-9a6a-7bc7bec58acb None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:21.059291 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe2c5c42-7b1c-462c-9a6a-7bc7bec58acb None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:21.071589 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe2c5c42-7b1c-462c-9a6a-7bc7bec58acb None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 13 15:53:21.077909 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application [None req-fe2c5c42-7b1c-462c-9a6a-7bc7bec58acb None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 13 15:53:21.077909 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:53:21.077909 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:53:21.077909 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:53:21.077909 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:53:21.077909 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:53:21.077909 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:53:21.077909 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:53:21.077909 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:53:21.077909 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:53:21.077909 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:53:21.077909 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:53:21.077909 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 13 15:53:21.077909 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 13 15:53:21.077909 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 13 15:53:21.077909 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 13 15:53:21.078811 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 13 15:53:21.078811 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 13 15:53:21.078811 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 13 15:53:21.078811 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 13 15:53:21.078811 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:53:21.078811 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:53:21.078811 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:53:21.078811 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:53:21.078811 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:53:21.078811 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:53:21.078811 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:53:21.078811 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application with Lock( Sep 13 15:53:21.078811 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:53:21.078811 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:53:21.078811 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:53:21.078811 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:53:21.079685 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:53:21.079685 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:53:21.079685 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:53:21.079685 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:53:21.079685 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 13 15:53:21.079685 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 13 15:53:21.079685 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 13 15:53:21.079685 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 13 15:53:21.079685 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 13 15:53:21.079685 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 13 15:53:21.079685 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 13 15:53:21.079685 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Sep 13 15:53:21.079685 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 107/222] 10.222.0.50 () {56 vars in 1096 bytes} [Tue Sep 13 15:53:20 2022] GET /identity/v3/roles/admin => generated 83 bytes in 188 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 13 15:53:21.081463 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c3f4336-798b-4383-b7ef-21efa9d29dd4 None None] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:21.081819 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c3f4336-798b-4383-b7ef-21efa9d29dd4 None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:21.082328 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c3f4336-798b-4383-b7ef-21efa9d29dd4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:21.087199 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb8d948c-cc43-4ff1-9dd1-70104db4a0cf None None] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:53:21.101621 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.auth.core [None req-5c3f4336-798b-4383-b7ef-21efa9d29dd4 None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81438) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:53:21.103182 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-5c3f4336-798b-4383-b7ef-21efa9d29dd4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:53:21.119396 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:53:21.127902 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdb0ed1c-067c-4102-9493-7406c81e0c53 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:53:21.128945 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdb0ed1c-067c-4102-9493-7406c81e0c53 None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:21.129402 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdb0ed1c-067c-4102-9493-7406c81e0c53 None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:21.129851 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdb0ed1c-067c-4102-9493-7406c81e0c53 None admin] PATH_INFO: `/v3/projects/9ec1adca55204df2b653531664b60123` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:21.131275 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cdb0ed1c-067c-4102-9493-7406c81e0c53 None admin] RBAC: Authorizing `identity:get_project(project_id=9ec1adca55204df2b653531664b60123)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:53:21.133201 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cdb0ed1c-067c-4102-9493-7406c81e0c53 None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:53:21.135194 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 116/223] 10.222.0.50 () {56 vars in 1186 bytes} [Tue Sep 13 15:53:20 2022] GET /identity/v3/projects/9ec1adca55204df2b653531664b60123 => generated 302 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 13 15:53:21.136904 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdb0ed1c-067c-4102-9493-7406c81e0c53 None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:53:21.181092 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 108/224] 10.222.0.50 () {58 vars in 1024 bytes} [Tue Sep 13 15:53:21 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 13 15:53:21.182884 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c3f4336-798b-4383-b7ef-21efa9d29dd4 None None] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:53:21.344637 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdb0ed1c-067c-4102-9493-7406c81e0c53 None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:53:21.361362 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c3f4336-798b-4383-b7ef-21efa9d29dd4 None None] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:53:21.398177 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8cf8ac4-7170-4dfc-b3e9-07acb9262811 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:53:21.399124 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8cf8ac4-7170-4dfc-b3e9-07acb9262811 None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:21.399898 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:53:21.400453 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8cf8ac4-7170-4dfc-b3e9-07acb9262811 None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:21.402105 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8cf8ac4-7170-4dfc-b3e9-07acb9262811 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:21.403142 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8cf8ac4-7170-4dfc-b3e9-07acb9262811 None admin] RBAC: Adding query filter params (role.id=15db4aaca3264ae29bde38903ba43471, scope.project.id=9ec1adca55204df2b653531664b60123, user.id=b8269ec40ba14c51825ec2be9c71b254) {{(pid=81438) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:53:21.403636 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8cf8ac4-7170-4dfc-b3e9-07acb9262811 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:53:21.405333 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8cf8ac4-7170-4dfc-b3e9-07acb9262811 None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:53:21.412885 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 109/225] 10.222.0.50 () {56 vars in 1374 bytes} [Tue Sep 13 15:53:21 2022] GET /identity/v3/role_assignments?user.id=b8269ec40ba14c51825ec2be9c71b254&scope.project.id=9ec1adca55204df2b653531664b60123&role.id=15db4aaca3264ae29bde38903ba43471 => generated 259 bytes in 231 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:53:21.415606 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fae2b29b-1bf6-44b4-87eb-04c6fe4d04cc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:53:21.416447 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fae2b29b-1bf6-44b4-87eb-04c6fe4d04cc None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:21.417463 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fae2b29b-1bf6-44b4-87eb-04c6fe4d04cc None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:21.417944 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fae2b29b-1bf6-44b4-87eb-04c6fe4d04cc None admin] PATH_INFO: `/v3/roles` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:21.418811 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fae2b29b-1bf6-44b4-87eb-04c6fe4d04cc None admin] RBAC: Adding query filter params (name=admin) {{(pid=81437) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:53:21.419504 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fae2b29b-1bf6-44b4-87eb-04c6fe4d04cc None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:53:21.421177 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fae2b29b-1bf6-44b4-87eb-04c6fe4d04cc None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:53:21.431806 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 117/226] 10.222.0.50 () {56 vars in 1099 bytes} [Tue Sep 13 15:53:21 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 296 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:53:21.440151 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8cf8ac4-7170-4dfc-b3e9-07acb9262811 None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:53:21.465583 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:53:21.487490 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fae2b29b-1bf6-44b4-87eb-04c6fe4d04cc None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:53:21.614232 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8cf8ac4-7170-4dfc-b3e9-07acb9262811 None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:53:21.615264 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fae2b29b-1bf6-44b4-87eb-04c6fe4d04cc None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:53:21.660628 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f66b252-07f8-4556-8c4e-5ab9525d2c9b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:53:21.660628 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f66b252-07f8-4556-8c4e-5ab9525d2c9b None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:21.660628 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f66b252-07f8-4556-8c4e-5ab9525d2c9b None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:21.660628 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f66b252-07f8-4556-8c4e-5ab9525d2c9b None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:21.674343 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f66b252-07f8-4556-8c4e-5ab9525d2c9b None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 13 15:53:21.680504 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application [None req-0f66b252-07f8-4556-8c4e-5ab9525d2c9b None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 13 15:53:21.680504 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:53:21.680504 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:53:21.680504 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:53:21.680504 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:53:21.680504 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:53:21.680504 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:53:21.680504 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:53:21.680504 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:53:21.680504 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:53:21.680504 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:53:21.680504 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:53:21.680504 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 13 15:53:21.680504 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 13 15:53:21.680504 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 13 15:53:21.680504 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 13 15:53:21.681643 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 13 15:53:21.681643 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 13 15:53:21.681643 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 13 15:53:21.681643 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 13 15:53:21.681643 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:53:21.681643 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:53:21.681643 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:53:21.681643 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:53:21.681643 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:53:21.681643 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:53:21.681643 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:53:21.681643 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application with Lock( Sep 13 15:53:21.681643 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:53:21.681643 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:53:21.681643 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:53:21.681643 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:53:21.683495 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:53:21.683495 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:53:21.683495 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:53:21.683495 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:53:21.683495 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 13 15:53:21.683495 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 13 15:53:21.683495 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 13 15:53:21.683495 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 13 15:53:21.683495 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 13 15:53:21.683495 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 13 15:53:21.683495 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 13 15:53:21.683495 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application Sep 13 15:53:21.687261 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e89ad397-d17e-4e24-b22b-c41c0fc9e42d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:53:21.688349 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e89ad397-d17e-4e24-b22b-c41c0fc9e42d None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:21.688766 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e89ad397-d17e-4e24-b22b-c41c0fc9e42d None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:21.694946 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 118/227] 10.222.0.50 () {56 vars in 1114 bytes} [Tue Sep 13 15:53:21 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 198 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 13 15:53:21.694946 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f66b252-07f8-4556-8c4e-5ab9525d2c9b None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:53:21.695679 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e89ad397-d17e-4e24-b22b-c41c0fc9e42d None admin] PATH_INFO: `/v3/users/ff3f9868e06e488a99e9902368f1a78a` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:21.702823 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e89ad397-d17e-4e24-b22b-c41c0fc9e42d None admin] RBAC: Authorizing `identity:get_user(user_id=ff3f9868e06e488a99e9902368f1a78a)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:53:21.704599 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e89ad397-d17e-4e24-b22b-c41c0fc9e42d None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:53:21.706571 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 110/228] 10.222.0.50 () {56 vars in 1177 bytes} [Tue Sep 13 15:53:21 2022] GET /identity/v3/users/ff3f9868e06e488a99e9902368f1a78a => generated 283 bytes in 267 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:53:21.721974 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e89ad397-d17e-4e24-b22b-c41c0fc9e42d None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:53:21.814363 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:53:21.848502 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f66b252-07f8-4556-8c4e-5ab9525d2c9b None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:53:21.867598 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:53:21.890257 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5eca3ad-b81d-44a1-9047-fab090d6b71b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:53:21.891373 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5eca3ad-b81d-44a1-9047-fab090d6b71b None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:21.891854 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5eca3ad-b81d-44a1-9047-fab090d6b71b None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:21.892377 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5eca3ad-b81d-44a1-9047-fab090d6b71b None admin] PATH_INFO: `/v3/roles` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:21.893243 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5eca3ad-b81d-44a1-9047-fab090d6b71b None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81437) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:53:21.893743 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5eca3ad-b81d-44a1-9047-fab090d6b71b None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:53:21.895441 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5eca3ad-b81d-44a1-9047-fab090d6b71b None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:53:21.904641 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 119/229] 10.222.0.50 () {56 vars in 1111 bytes} [Tue Sep 13 15:53:21 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 214 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 13 15:53:21.908987 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e89ad397-d17e-4e24-b22b-c41c0fc9e42d None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:53:21.910744 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5eca3ad-b81d-44a1-9047-fab090d6b71b None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:53:21.956424 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1d5a0ff-bec5-4489-ba0f-de03f2e9de1e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:53:21.957065 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1d5a0ff-bec5-4489-ba0f-de03f2e9de1e None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:21.957349 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1d5a0ff-bec5-4489-ba0f-de03f2e9de1e None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:21.957624 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1d5a0ff-bec5-4489-ba0f-de03f2e9de1e None admin] PATH_INFO: `/v3/projects/9ec1adca55204df2b653531664b60123` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:21.958769 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1d5a0ff-bec5-4489-ba0f-de03f2e9de1e None admin] RBAC: Authorizing `identity:get_project(project_id=9ec1adca55204df2b653531664b60123)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:53:21.960244 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1d5a0ff-bec5-4489-ba0f-de03f2e9de1e None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:53:21.961886 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 111/230] 10.222.0.50 () {56 vars in 1186 bytes} [Tue Sep 13 15:53:21 2022] GET /identity/v3/projects/9ec1adca55204df2b653531664b60123 => generated 302 bytes in 240 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:53:21.973464 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1d5a0ff-bec5-4489-ba0f-de03f2e9de1e None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:53:22.083642 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5eca3ad-b81d-44a1-9047-fab090d6b71b None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:53:22.101869 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1d5a0ff-bec5-4489-ba0f-de03f2e9de1e None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:53:22.119326 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:53:22.144622 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31414911-639a-44cc-8a9a-4ce1d67afcb5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:53:22.145286 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31414911-639a-44cc-8a9a-4ce1d67afcb5 None admin] REQUEST_METHOD: `PUT` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:22.145564 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31414911-639a-44cc-8a9a-4ce1d67afcb5 None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:22.145836 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31414911-639a-44cc-8a9a-4ce1d67afcb5 None admin] PATH_INFO: `/v3/projects/9ec1adca55204df2b653531664b60123/users/ff3f9868e06e488a99e9902368f1a78a/roles/031d4e1a23e34f71858fc4a6dfbd4ccb` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:22.148734 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31414911-639a-44cc-8a9a-4ce1d67afcb5 None admin] RBAC: Authorizing `identity:create_grant(project_id=9ec1adca55204df2b653531664b60123, user_id=ff3f9868e06e488a99e9902368f1a78a, role_id=031d4e1a23e34f71858fc4a6dfbd4ccb)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:53:22.150170 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31414911-639a-44cc-8a9a-4ce1d67afcb5 None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:53:22.162019 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e071fe4e-ca6a-48fb-b8cf-b10586a68114 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:53:22.162713 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e071fe4e-ca6a-48fb-b8cf-b10586a68114 None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:22.163049 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e071fe4e-ca6a-48fb-b8cf-b10586a68114 None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:22.163358 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e071fe4e-ca6a-48fb-b8cf-b10586a68114 None admin] PATH_INFO: `/v3/users/ff3f9868e06e488a99e9902368f1a78a` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:22.165270 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e071fe4e-ca6a-48fb-b8cf-b10586a68114 None admin] RBAC: Authorizing `identity:get_user(user_id=ff3f9868e06e488a99e9902368f1a78a)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:53:22.170057 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e071fe4e-ca6a-48fb-b8cf-b10586a68114 None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:53:22.174969 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 112/231] 10.222.0.50 () {58 vars in 1439 bytes} [Tue Sep 13 15:53:21 2022] PUT /identity/v3/projects/9ec1adca55204df2b653531664b60123/users/ff3f9868e06e488a99e9902368f1a78a/roles/031d4e1a23e34f71858fc4a6dfbd4ccb => generated 0 bytes in 202 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 13 15:53:22.175228 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 120/232] 10.222.0.50 () {56 vars in 1177 bytes} [Tue Sep 13 15:53:21 2022] GET /identity/v3/users/ff3f9868e06e488a99e9902368f1a78a => generated 283 bytes in 265 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:53:22.178394 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31414911-639a-44cc-8a9a-4ce1d67afcb5 None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:53:22.193972 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:53:22.332034 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31414911-639a-44cc-8a9a-4ce1d67afcb5 None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:53:22.393393 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d731662-4d23-4fd0-b5e4-2edf52e231f0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:53:22.394405 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d731662-4d23-4fd0-b5e4-2edf52e231f0 None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:22.394940 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d731662-4d23-4fd0-b5e4-2edf52e231f0 None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:22.395420 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d731662-4d23-4fd0-b5e4-2edf52e231f0 None admin] PATH_INFO: `/v3/projects/9ec1adca55204df2b653531664b60123` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:22.396873 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d731662-4d23-4fd0-b5e4-2edf52e231f0 None admin] RBAC: Authorizing `identity:get_project(project_id=9ec1adca55204df2b653531664b60123)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:53:22.398560 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d731662-4d23-4fd0-b5e4-2edf52e231f0 None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:53:22.400526 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 113/233] 10.222.0.50 () {56 vars in 1186 bytes} [Tue Sep 13 15:53:22 2022] GET /identity/v3/projects/9ec1adca55204df2b653531664b60123 => generated 302 bytes in 222 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:53:22.406591 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e071fe4e-ca6a-48fb-b8cf-b10586a68114 None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:53:22.556478 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e071fe4e-ca6a-48fb-b8cf-b10586a68114 None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:53:22.613713 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3af8e90-b1bc-474d-a34b-5e2430da0920 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:53:22.614417 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3af8e90-b1bc-474d-a34b-5e2430da0920 None admin] REQUEST_METHOD: `PUT` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:22.614696 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3af8e90-b1bc-474d-a34b-5e2430da0920 None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:22.615002 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3af8e90-b1bc-474d-a34b-5e2430da0920 None admin] PATH_INFO: `/v3/projects/9ec1adca55204df2b653531664b60123/users/ff3f9868e06e488a99e9902368f1a78a/roles/f74e3f6c20bb426ca030f211c3a9aef5` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:22.617585 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3af8e90-b1bc-474d-a34b-5e2430da0920 None admin] RBAC: Authorizing `identity:create_grant(project_id=9ec1adca55204df2b653531664b60123, user_id=ff3f9868e06e488a99e9902368f1a78a, role_id=f74e3f6c20bb426ca030f211c3a9aef5)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:53:22.619206 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3af8e90-b1bc-474d-a34b-5e2430da0920 None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:53:22.649188 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 121/234] 10.222.0.50 () {58 vars in 1439 bytes} [Tue Sep 13 15:53:22 2022] PUT /identity/v3/projects/9ec1adca55204df2b653531664b60123/users/ff3f9868e06e488a99e9902368f1a78a/roles/f74e3f6c20bb426ca030f211c3a9aef5 => generated 0 bytes in 243 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:53:23.007695 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4405d20-35b5-4b70-a848-d8192548c3fd None None] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:23.008685 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4405d20-35b5-4b70-a848-d8192548c3fd None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:23.009175 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4405d20-35b5-4b70-a848-d8192548c3fd None None] PATH_INFO: `/` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:23.011539 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 114/235] 10.222.0.50 () {54 vars in 927 bytes} [Tue Sep 13 15:53:23 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:53:23.018962 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c511aac-c28d-49c1-aa40-c3d81bbebd48 None None] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:23.019246 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c511aac-c28d-49c1-aa40-c3d81bbebd48 None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:23.019517 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c511aac-c28d-49c1-aa40-c3d81bbebd48 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:23.043207 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.auth.core [None req-2c511aac-c28d-49c1-aa40-c3d81bbebd48 None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81437) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:53:23.044427 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-2c511aac-c28d-49c1-aa40-c3d81bbebd48 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:53:23.092887 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:53:23.153550 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 122/236] 10.222.0.50 () {58 vars in 1024 bytes} [Tue Sep 13 15:53:23 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 136 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 13 15:53:23.214473 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-345dcdd6-0d61-4590-909b-81ae06d87223 None None] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:23.215028 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-345dcdd6-0d61-4590-909b-81ae06d87223 None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:23.215477 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-345dcdd6-0d61-4590-909b-81ae06d87223 None None] PATH_INFO: `/` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:23.217231 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 115/237] 10.222.0.50 () {54 vars in 927 bytes} [Tue Sep 13 15:53:23 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:53:23.222708 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-121c54f7-7293-4bed-a32f-040b0c5be445 None None] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:23.223027 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-121c54f7-7293-4bed-a32f-040b0c5be445 None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:23.223304 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-121c54f7-7293-4bed-a32f-040b0c5be445 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:23.247954 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.auth.core [None req-121c54f7-7293-4bed-a32f-040b0c5be445 None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81437) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:53:23.249172 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-121c54f7-7293-4bed-a32f-040b0c5be445 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:53:23.343782 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 123/238] 10.222.0.50 () {58 vars in 1024 bytes} [Tue Sep 13 15:53:23 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 122 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 13 15:53:23.472963 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-345dcdd6-0d61-4590-909b-81ae06d87223 None None] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:53:23.570962 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:53:23.597816 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-345dcdd6-0d61-4590-909b-81ae06d87223 None None] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:53:23.643179 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-539b439f-8161-4b98-8c04-594750d27dd3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:53:23.643822 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-539b439f-8161-4b98-8c04-594750d27dd3 None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:23.644105 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-539b439f-8161-4b98-8c04-594750d27dd3 None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:23.644380 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-539b439f-8161-4b98-8c04-594750d27dd3 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:23.661470 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-539b439f-8161-4b98-8c04-594750d27dd3 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 13 15:53:23.666389 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application [None req-539b439f-8161-4b98-8c04-594750d27dd3 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Sep 13 15:53:23.666389 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:53:23.666389 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:53:23.666389 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:53:23.666389 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:53:23.666389 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:53:23.666389 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:53:23.666389 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:53:23.666389 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:53:23.666389 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:53:23.666389 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:53:23.666389 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:53:23.666389 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 13 15:53:23.666389 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 13 15:53:23.666389 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 13 15:53:23.666389 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 13 15:53:23.667646 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 13 15:53:23.667646 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 13 15:53:23.667646 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 13 15:53:23.667646 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 13 15:53:23.667646 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:53:23.667646 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:53:23.667646 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:53:23.667646 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:53:23.667646 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:53:23.667646 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:53:23.667646 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:53:23.667646 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application with Lock( Sep 13 15:53:23.667646 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:53:23.667646 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:53:23.667646 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:53:23.667646 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:53:23.668443 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:53:23.668443 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:53:23.668443 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:53:23.668443 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:53:23.668443 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 13 15:53:23.668443 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 13 15:53:23.668443 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 13 15:53:23.668443 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 13 15:53:23.668443 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 13 15:53:23.668443 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 13 15:53:23.668443 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Sep 13 15:53:23.668443 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Sep 13 15:53:23.668443 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 116/239] 10.222.0.50 () {56 vars in 1099 bytes} [Tue Sep 13 15:53:23 2022] GET /identity/v3/roles/reader => generated 84 bytes in 195 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:53:23.674094 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-121c54f7-7293-4bed-a32f-040b0c5be445 None None] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:53:23.724637 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-539b439f-8161-4b98-8c04-594750d27dd3 None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:53:23.857145 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:53:23.861590 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-121c54f7-7293-4bed-a32f-040b0c5be445 None None] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:53:23.880695 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-539b439f-8161-4b98-8c04-594750d27dd3 None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:53:23.930351 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90ba0b03-4e87-4985-9c7a-5aeff7977cd1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:53:23.930999 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90ba0b03-4e87-4985-9c7a-5aeff7977cd1 None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:23.931283 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90ba0b03-4e87-4985-9c7a-5aeff7977cd1 None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:23.931558 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90ba0b03-4e87-4985-9c7a-5aeff7977cd1 None admin] PATH_INFO: `/v3/roles` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:23.933125 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90ba0b03-4e87-4985-9c7a-5aeff7977cd1 None admin] RBAC: Adding query filter params (name=reader) {{(pid=81437) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:53:23.933674 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90ba0b03-4e87-4985-9c7a-5aeff7977cd1 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:53:23.936576 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90ba0b03-4e87-4985-9c7a-5aeff7977cd1 None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:53:23.950797 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c04d4973-6f76-4509-8139-5d725842b7ae None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:53:23.951486 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c04d4973-6f76-4509-8139-5d725842b7ae None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:23.951765 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c04d4973-6f76-4509-8139-5d725842b7ae None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:23.952035 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c04d4973-6f76-4509-8139-5d725842b7ae None admin] PATH_INFO: `/v3/domains/default` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:23.953491 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c04d4973-6f76-4509-8139-5d725842b7ae None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:53:23.955059 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c04d4973-6f76-4509-8139-5d725842b7ae None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:53:23.963337 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 124/240] 10.222.0.50 () {56 vars in 1101 bytes} [Tue Sep 13 15:53:23 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 286 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:53:23.964979 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90ba0b03-4e87-4985-9c7a-5aeff7977cd1 None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:53:23.965636 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 117/241] 10.222.0.50 () {56 vars in 1108 bytes} [Tue Sep 13 15:53:23 2022] GET /identity/v3/domains/default => generated 199 bytes in 241 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:53:23.966505 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c04d4973-6f76-4509-8139-5d725842b7ae None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:53:23.973088 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:53:24.089696 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90ba0b03-4e87-4985-9c7a-5aeff7977cd1 None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:53:24.127056 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7146c77-0b02-4850-82da-de9ab26b65db None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:53:24.127927 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7146c77-0b02-4850-82da-de9ab26b65db None admin] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:24.128355 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7146c77-0b02-4850-82da-de9ab26b65db None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:24.128768 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7146c77-0b02-4850-82da-de9ab26b65db None admin] PATH_INFO: `/v3/users` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:24.129574 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7146c77-0b02-4850-82da-de9ab26b65db None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:53:24.131182 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7146c77-0b02-4850-82da-de9ab26b65db None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:53:24.153822 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 125/242] 10.222.0.50 () {60 vars in 1132 bytes} [Tue Sep 13 15:53:23 2022] POST /identity/v3/users => generated 293 bytes in 190 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 15:53:24.155693 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f46e789-3863-4524-92dd-5adf04dc2105 None None] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:24.156119 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f46e789-3863-4524-92dd-5adf04dc2105 None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:24.156571 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f46e789-3863-4524-92dd-5adf04dc2105 None None] PATH_INFO: `/` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:24.158577 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 126/243] 10.222.0.50 () {54 vars in 927 bytes} [Tue Sep 13 15:53:24 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:53:24.165322 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c04d4973-6f76-4509-8139-5d725842b7ae None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:53:24.170637 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5ca0f7c-a048-4469-a46b-a7e6c835fdf3 None None] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:24.171279 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5ca0f7c-a048-4469-a46b-a7e6c835fdf3 None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:24.171749 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5ca0f7c-a048-4469-a46b-a7e6c835fdf3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:24.201505 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.auth.core [None req-c5ca0f7c-a048-4469-a46b-a7e6c835fdf3 None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81437) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:53:24.203844 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-c5ca0f7c-a048-4469-a46b-a7e6c835fdf3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:53:24.224986 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28dbab88-8077-4203-a187-2535a3c64128 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:53:24.226479 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28dbab88-8077-4203-a187-2535a3c64128 None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:24.227169 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28dbab88-8077-4203-a187-2535a3c64128 None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:24.229904 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28dbab88-8077-4203-a187-2535a3c64128 None admin] PATH_INFO: `/v3/users/4f274b7d64b642e39dd6be3b8cf368df` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:24.241311 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:53:24.268574 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28dbab88-8077-4203-a187-2535a3c64128 None admin] RBAC: Authorizing `identity:get_user(user_id=4f274b7d64b642e39dd6be3b8cf368df)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:53:24.270601 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28dbab88-8077-4203-a187-2535a3c64128 None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:53:24.274668 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 118/244] 10.222.0.50 () {56 vars in 1177 bytes} [Tue Sep 13 15:53:23 2022] GET /identity/v3/users/4f274b7d64b642e39dd6be3b8cf368df => generated 297 bytes in 308 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:53:24.287973 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28dbab88-8077-4203-a187-2535a3c64128 None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:53:24.365478 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 127/245] 10.222.0.50 () {58 vars in 1024 bytes} [Tue Sep 13 15:53:24 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 196 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 13 15:53:24.375836 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:53:24.418336 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28dbab88-8077-4203-a187-2535a3c64128 None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:53:24.453044 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d79b59f-2076-4b21-a0f9-5293db708645 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:53:24.453659 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d79b59f-2076-4b21-a0f9-5293db708645 None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:24.453938 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d79b59f-2076-4b21-a0f9-5293db708645 None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:24.454215 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d79b59f-2076-4b21-a0f9-5293db708645 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:24.524693 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d79b59f-2076-4b21-a0f9-5293db708645 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:53:24.527043 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d79b59f-2076-4b21-a0f9-5293db708645 None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:53:24.588735 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:53:24.634054 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 119/246] 10.222.0.50 () {58 vars in 1282 bytes} [Tue Sep 13 15:53:24 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 343 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 13 15:53:24.636772 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5ca0f7c-a048-4469-a46b-a7e6c835fdf3 None None] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:53:24.717099 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d79b59f-2076-4b21-a0f9-5293db708645 None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:53:24.754301 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5ca0f7c-a048-4469-a46b-a7e6c835fdf3 None None] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:53:24.801998 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e9798c7-0637-480a-b404-cf838231aaaa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:53:24.802679 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e9798c7-0637-480a-b404-cf838231aaaa None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:24.802987 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e9798c7-0637-480a-b404-cf838231aaaa None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:24.803265 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e9798c7-0637-480a-b404-cf838231aaaa None admin] PATH_INFO: `/v3/projects/9ec1adca55204df2b653531664b60123` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:24.805067 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e9798c7-0637-480a-b404-cf838231aaaa None admin] RBAC: Authorizing `identity:get_project(project_id=9ec1adca55204df2b653531664b60123)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:53:24.806704 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e9798c7-0637-480a-b404-cf838231aaaa None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:53:24.809774 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 128/247] 10.222.0.50 () {56 vars in 1186 bytes} [Tue Sep 13 15:53:24 2022] GET /identity/v3/projects/9ec1adca55204df2b653531664b60123 => generated 302 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:53:24.817166 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e9798c7-0637-480a-b404-cf838231aaaa None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:53:24.843762 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:53:24.857736 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d79b59f-2076-4b21-a0f9-5293db708645 None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:53:24.911140 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17638e29-cf93-4ae8-8011-8d36d5a151eb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:53:24.911784 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17638e29-cf93-4ae8-8011-8d36d5a151eb None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:24.912061 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17638e29-cf93-4ae8-8011-8d36d5a151eb None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:24.912318 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17638e29-cf93-4ae8-8011-8d36d5a151eb None admin] PATH_INFO: `/v3/roles/member` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:24.925057 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17638e29-cf93-4ae8-8011-8d36d5a151eb None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 13 15:53:24.932409 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application [None req-17638e29-cf93-4ae8-8011-8d36d5a151eb None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 13 15:53:24.932409 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:53:24.932409 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:53:24.932409 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:53:24.932409 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:53:24.932409 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:53:24.932409 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:53:24.932409 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:53:24.932409 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:53:24.932409 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:53:24.932409 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:53:24.932409 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:53:24.932409 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 13 15:53:24.932409 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 13 15:53:24.932409 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 13 15:53:24.932409 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 13 15:53:24.933882 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 13 15:53:24.933882 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 13 15:53:24.933882 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 13 15:53:24.933882 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 13 15:53:24.933882 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:53:24.933882 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:53:24.933882 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:53:24.933882 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:53:24.933882 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:53:24.933882 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:53:24.933882 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:53:24.933882 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application with Lock( Sep 13 15:53:24.933882 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:53:24.933882 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:53:24.933882 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:53:24.933882 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:53:24.935068 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:53:24.935068 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:53:24.935068 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:53:24.935068 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:53:24.935068 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 13 15:53:24.935068 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 13 15:53:24.935068 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 13 15:53:24.935068 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 13 15:53:24.935068 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 13 15:53:24.935068 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 13 15:53:24.935068 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 13 15:53:24.935068 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Sep 13 15:53:24.935068 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 120/248] 10.222.0.50 () {56 vars in 1099 bytes} [Tue Sep 13 15:53:24 2022] GET /identity/v3/roles/member => generated 84 bytes in 216 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 13 15:53:24.939547 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17638e29-cf93-4ae8-8011-8d36d5a151eb None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:53:24.949510 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e9798c7-0637-480a-b404-cf838231aaaa None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:53:24.970980 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:53:24.988701 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e962fcde-9920-414e-b6b0-d8c292f5df43 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:53:24.989659 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e962fcde-9920-414e-b6b0-d8c292f5df43 None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:24.990180 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e962fcde-9920-414e-b6b0-d8c292f5df43 None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:24.990654 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e962fcde-9920-414e-b6b0-d8c292f5df43 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:24.991611 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e962fcde-9920-414e-b6b0-d8c292f5df43 None admin] RBAC: Adding query filter params (role.id=b2d1670446a54085ad2bdb935f003d25, scope.project.id=9ec1adca55204df2b653531664b60123, user.id=4f274b7d64b642e39dd6be3b8cf368df) {{(pid=81437) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:53:24.992124 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e962fcde-9920-414e-b6b0-d8c292f5df43 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:53:24.993760 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e962fcde-9920-414e-b6b0-d8c292f5df43 None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:53:25.007057 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 129/249] 10.222.0.50 () {56 vars in 1374 bytes} [Tue Sep 13 15:53:24 2022] GET /identity/v3/role_assignments?user.id=4f274b7d64b642e39dd6be3b8cf368df&scope.project.id=9ec1adca55204df2b653531664b60123&role.id=b2d1670446a54085ad2bdb935f003d25 => generated 259 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:53:25.062581 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17638e29-cf93-4ae8-8011-8d36d5a151eb None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:53:25.110214 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d58469ec-ad1a-4134-95d2-d2118614aa40 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:53:25.110891 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d58469ec-ad1a-4134-95d2-d2118614aa40 None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:25.111197 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d58469ec-ad1a-4134-95d2-d2118614aa40 None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:25.111469 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d58469ec-ad1a-4134-95d2-d2118614aa40 None admin] PATH_INFO: `/v3/roles` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:25.112144 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d58469ec-ad1a-4134-95d2-d2118614aa40 None admin] RBAC: Adding query filter params (name=member) {{(pid=81438) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:53:25.112528 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d58469ec-ad1a-4134-95d2-d2118614aa40 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:53:25.113965 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d58469ec-ad1a-4134-95d2-d2118614aa40 None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:53:25.124175 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 121/250] 10.222.0.50 () {56 vars in 1101 bytes} [Tue Sep 13 15:53:24 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:53:25.129787 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e962fcde-9920-414e-b6b0-d8c292f5df43 None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:53:25.197308 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3d6dc1e-84bb-434b-ab81-5a2c718e3799 None None] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:25.197815 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3d6dc1e-84bb-434b-ab81-5a2c718e3799 None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:25.198092 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3d6dc1e-84bb-434b-ab81-5a2c718e3799 None None] PATH_INFO: `/` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:25.199659 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 122/251] 10.222.0.50 () {54 vars in 927 bytes} [Tue Sep 13 15:53:25 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 13 15:53:25.205443 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-479ee7c4-a5b9-4e2f-9da1-1e792d6d050a None None] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:25.205725 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-479ee7c4-a5b9-4e2f-9da1-1e792d6d050a None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:25.205990 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-479ee7c4-a5b9-4e2f-9da1-1e792d6d050a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:25.227312 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.auth.core [None req-479ee7c4-a5b9-4e2f-9da1-1e792d6d050a None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81438) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:53:25.228453 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-479ee7c4-a5b9-4e2f-9da1-1e792d6d050a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:53:25.247945 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:53:25.285748 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e962fcde-9920-414e-b6b0-d8c292f5df43 None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:53:25.314813 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 123/252] 10.222.0.50 () {58 vars in 1024 bytes} [Tue Sep 13 15:53:25 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 13 15:53:25.338411 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1549cff6-65a5-4737-8890-01ad964fdea9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:53:25.341687 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1549cff6-65a5-4737-8890-01ad964fdea9 None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:25.342165 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1549cff6-65a5-4737-8890-01ad964fdea9 None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:25.342575 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1549cff6-65a5-4737-8890-01ad964fdea9 None admin] PATH_INFO: `/v3/users/b8269ec40ba14c51825ec2be9c71b254` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:25.347783 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1549cff6-65a5-4737-8890-01ad964fdea9 None admin] RBAC: Authorizing `identity:get_user(user_id=b8269ec40ba14c51825ec2be9c71b254)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:53:25.349522 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1549cff6-65a5-4737-8890-01ad964fdea9 None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:53:25.351806 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 130/253] 10.222.0.50 () {56 vars in 1177 bytes} [Tue Sep 13 15:53:25 2022] GET /identity/v3/users/b8269ec40ba14c51825ec2be9c71b254 => generated 297 bytes in 222 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:53:25.358475 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-479ee7c4-a5b9-4e2f-9da1-1e792d6d050a None None] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:53:25.383299 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:53:25.478840 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-479ee7c4-a5b9-4e2f-9da1-1e792d6d050a None None] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:53:25.507715 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99ffe5a8-b88f-4eba-a977-2480d68f840c None None] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:25.508016 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99ffe5a8-b88f-4eba-a977-2480d68f840c None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:25.508257 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99ffe5a8-b88f-4eba-a977-2480d68f840c None None] PATH_INFO: `/` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:25.509738 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 131/254] 10.222.0.50 () {54 vars in 927 bytes} [Tue Sep 13 15:53:25 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:53:25.515472 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f79c4f63-8589-40ae-909c-1c6eba21cada None None] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:25.516039 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f79c4f63-8589-40ae-909c-1c6eba21cada None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:25.516511 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f79c4f63-8589-40ae-909c-1c6eba21cada None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:25.532146 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c057c174-4175-47dd-a89e-46a11afee19d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:53:25.533150 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c057c174-4175-47dd-a89e-46a11afee19d None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:25.533636 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c057c174-4175-47dd-a89e-46a11afee19d None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:25.536218 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c057c174-4175-47dd-a89e-46a11afee19d None admin] PATH_INFO: `/v3/projects/9ec1adca55204df2b653531664b60123` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:25.537649 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c057c174-4175-47dd-a89e-46a11afee19d None admin] RBAC: Authorizing `identity:get_project(project_id=9ec1adca55204df2b653531664b60123)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:53:25.540751 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c057c174-4175-47dd-a89e-46a11afee19d None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:53:25.544019 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 124/255] 10.222.0.50 () {56 vars in 1186 bytes} [Tue Sep 13 15:53:25 2022] GET /identity/v3/projects/9ec1adca55204df2b653531664b60123 => generated 302 bytes in 186 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:53:25.547470 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.auth.core [None req-f79c4f63-8589-40ae-909c-1c6eba21cada None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81437) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:53:25.549531 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c057c174-4175-47dd-a89e-46a11afee19d None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:53:25.550632 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-f79c4f63-8589-40ae-909c-1c6eba21cada None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:53:25.632116 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 132/256] 10.222.0.50 () {58 vars in 1024 bytes} [Tue Sep 13 15:53:25 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 117 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 13 15:53:25.714503 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:53:25.738962 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c057c174-4175-47dd-a89e-46a11afee19d None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:53:25.787606 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b354f765-3858-4b78-be7d-01e91801e9c2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:53:25.788290 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b354f765-3858-4b78-be7d-01e91801e9c2 None admin] REQUEST_METHOD: `PUT` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:25.788573 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b354f765-3858-4b78-be7d-01e91801e9c2 None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:25.788846 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b354f765-3858-4b78-be7d-01e91801e9c2 None admin] PATH_INFO: `/v3/projects/9ec1adca55204df2b653531664b60123/users/b8269ec40ba14c51825ec2be9c71b254/roles/15db4aaca3264ae29bde38903ba43471` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:25.790833 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b354f765-3858-4b78-be7d-01e91801e9c2 None admin] RBAC: Authorizing `identity:create_grant(project_id=9ec1adca55204df2b653531664b60123, user_id=b8269ec40ba14c51825ec2be9c71b254, role_id=15db4aaca3264ae29bde38903ba43471)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:53:25.792339 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b354f765-3858-4b78-be7d-01e91801e9c2 None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:53:25.814159 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 125/257] 10.222.0.50 () {58 vars in 1439 bytes} [Tue Sep 13 15:53:25 2022] PUT /identity/v3/projects/9ec1adca55204df2b653531664b60123/users/b8269ec40ba14c51825ec2be9c71b254/roles/15db4aaca3264ae29bde38903ba43471 => generated 0 bytes in 265 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:53:25.851052 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f79c4f63-8589-40ae-909c-1c6eba21cada None None] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:53:25.967608 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:53:25.992904 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f79c4f63-8589-40ae-909c-1c6eba21cada None None] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:53:26.021013 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b354f765-3858-4b78-be7d-01e91801e9c2 None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:53:26.045989 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58e914fa-504f-4d54-baf0-aa9c55f8b2d7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:53:26.046652 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58e914fa-504f-4d54-baf0-aa9c55f8b2d7 None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:26.046956 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58e914fa-504f-4d54-baf0-aa9c55f8b2d7 None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:26.047232 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58e914fa-504f-4d54-baf0-aa9c55f8b2d7 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:26.082392 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58e914fa-504f-4d54-baf0-aa9c55f8b2d7 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 13 15:53:26.088322 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application [None req-58e914fa-504f-4d54-baf0-aa9c55f8b2d7 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 13 15:53:26.088322 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:53:26.088322 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:53:26.088322 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:53:26.088322 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:53:26.088322 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:53:26.088322 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:53:26.088322 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:53:26.088322 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:53:26.088322 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:53:26.088322 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:53:26.088322 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:53:26.088322 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 13 15:53:26.088322 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 13 15:53:26.088322 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 13 15:53:26.088322 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 13 15:53:26.090004 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 13 15:53:26.090004 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 13 15:53:26.090004 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 13 15:53:26.090004 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 13 15:53:26.090004 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:53:26.090004 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:53:26.090004 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:53:26.090004 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:53:26.090004 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:53:26.090004 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:53:26.090004 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:53:26.090004 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application with Lock( Sep 13 15:53:26.090004 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:53:26.090004 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:53:26.090004 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:53:26.090004 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:53:26.090810 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:53:26.090810 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:53:26.090810 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:53:26.090810 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:53:26.090810 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 13 15:53:26.090810 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 13 15:53:26.090810 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 13 15:53:26.090810 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 13 15:53:26.090810 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 13 15:53:26.090810 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 13 15:53:26.090810 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 13 15:53:26.090810 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application Sep 13 15:53:26.091454 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 133/258] 10.222.0.50 () {56 vars in 1096 bytes} [Tue Sep 13 15:53:25 2022] GET /identity/v3/roles/admin => generated 83 bytes in 240 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:53:26.099992 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58e914fa-504f-4d54-baf0-aa9c55f8b2d7 None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:53:26.201434 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b354f765-3858-4b78-be7d-01e91801e9c2 None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:53:26.247981 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f13ff878-e1da-4c51-9fed-3a7ceeca4976 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:53:26.248561 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f13ff878-e1da-4c51-9fed-3a7ceeca4976 None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:26.248838 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f13ff878-e1da-4c51-9fed-3a7ceeca4976 None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:26.249182 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f13ff878-e1da-4c51-9fed-3a7ceeca4976 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:26.263307 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58e914fa-504f-4d54-baf0-aa9c55f8b2d7 None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:53:26.267404 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f13ff878-e1da-4c51-9fed-3a7ceeca4976 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 13 15:53:26.279568 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application [None req-f13ff878-e1da-4c51-9fed-3a7ceeca4976 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 13 15:53:26.279568 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:53:26.279568 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:53:26.279568 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:53:26.279568 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:53:26.279568 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:53:26.279568 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:53:26.279568 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:53:26.279568 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:53:26.279568 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:53:26.279568 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:53:26.279568 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:53:26.279568 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 13 15:53:26.279568 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 13 15:53:26.279568 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 13 15:53:26.279568 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 13 15:53:26.280531 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 13 15:53:26.280531 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 13 15:53:26.280531 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 13 15:53:26.280531 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 13 15:53:26.280531 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:53:26.280531 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:53:26.280531 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:53:26.280531 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:53:26.280531 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:53:26.280531 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:53:26.280531 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:53:26.280531 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application with Lock( Sep 13 15:53:26.280531 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:53:26.280531 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:53:26.280531 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:53:26.280531 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:53:26.281318 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:53:26.281318 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:53:26.281318 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:53:26.281318 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:53:26.281318 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 13 15:53:26.281318 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 13 15:53:26.281318 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 13 15:53:26.281318 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 13 15:53:26.281318 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 13 15:53:26.281318 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 13 15:53:26.281318 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 13 15:53:26.281318 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Sep 13 15:53:26.281318 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 126/259] 10.222.0.50 () {56 vars in 1114 bytes} [Tue Sep 13 15:53:26 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 260 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:53:26.286693 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f13ff878-e1da-4c51-9fed-3a7ceeca4976 None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:53:26.290100 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:53:26.327627 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4276d58-0c04-4827-8f48-6a98b4564ecb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:53:26.328260 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4276d58-0c04-4827-8f48-6a98b4564ecb None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:26.328542 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4276d58-0c04-4827-8f48-6a98b4564ecb None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:26.328818 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4276d58-0c04-4827-8f48-6a98b4564ecb None admin] PATH_INFO: `/v3/roles` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:26.329515 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4276d58-0c04-4827-8f48-6a98b4564ecb None admin] RBAC: Adding query filter params (name=admin) {{(pid=81437) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:53:26.329811 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4276d58-0c04-4827-8f48-6a98b4564ecb None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:53:26.331233 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4276d58-0c04-4827-8f48-6a98b4564ecb None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:53:26.336769 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:53:26.351029 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 134/260] 10.222.0.50 () {56 vars in 1099 bytes} [Tue Sep 13 15:53:26 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 247 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 13 15:53:26.364029 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4276d58-0c04-4827-8f48-6a98b4564ecb None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:53:26.508752 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4276d58-0c04-4827-8f48-6a98b4564ecb None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:53:26.556117 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0891cbd0-cb57-4ec9-8872-66b0b56c03e1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:53:26.556761 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0891cbd0-cb57-4ec9-8872-66b0b56c03e1 None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:26.557352 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0891cbd0-cb57-4ec9-8872-66b0b56c03e1 None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:26.557664 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0891cbd0-cb57-4ec9-8872-66b0b56c03e1 None admin] PATH_INFO: `/v3/users/ff3f9868e06e488a99e9902368f1a78a` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:26.563588 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0891cbd0-cb57-4ec9-8872-66b0b56c03e1 None admin] RBAC: Authorizing `identity:get_user(user_id=ff3f9868e06e488a99e9902368f1a78a)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:53:26.565051 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0891cbd0-cb57-4ec9-8872-66b0b56c03e1 None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:53:26.566648 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 135/261] 10.222.0.50 () {56 vars in 1177 bytes} [Tue Sep 13 15:53:26 2022] GET /identity/v3/users/ff3f9868e06e488a99e9902368f1a78a => generated 283 bytes in 203 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:53:26.572472 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0891cbd0-cb57-4ec9-8872-66b0b56c03e1 None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:53:26.601744 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f13ff878-e1da-4c51-9fed-3a7ceeca4976 None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:53:26.680356 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:53:26.700399 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-beef1294-e19d-4b26-b84c-25336e992e27 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:53:26.701031 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-beef1294-e19d-4b26-b84c-25336e992e27 None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:26.701311 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-beef1294-e19d-4b26-b84c-25336e992e27 None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:26.701587 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-beef1294-e19d-4b26-b84c-25336e992e27 None admin] PATH_INFO: `/v3/roles` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:26.702257 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-beef1294-e19d-4b26-b84c-25336e992e27 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81438) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:53:26.702563 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-beef1294-e19d-4b26-b84c-25336e992e27 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:53:26.708092 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-beef1294-e19d-4b26-b84c-25336e992e27 None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:53:26.722404 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 127/262] 10.222.0.50 () {56 vars in 1111 bytes} [Tue Sep 13 15:53:26 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 436 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:53:26.727868 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-beef1294-e19d-4b26-b84c-25336e992e27 None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:53:26.737718 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0891cbd0-cb57-4ec9-8872-66b0b56c03e1 None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:53:26.786060 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66a3fe2b-d6e4-4a82-b9ac-30d05c195309 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:53:26.786615 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66a3fe2b-d6e4-4a82-b9ac-30d05c195309 None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:26.786873 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66a3fe2b-d6e4-4a82-b9ac-30d05c195309 None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:26.787186 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66a3fe2b-d6e4-4a82-b9ac-30d05c195309 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:26.848891 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:53:26.886111 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66a3fe2b-d6e4-4a82-b9ac-30d05c195309 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:53:26.887713 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66a3fe2b-d6e4-4a82-b9ac-30d05c195309 None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:53:26.929773 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-beef1294-e19d-4b26-b84c-25336e992e27 None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:53:26.957111 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:53:27.008621 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53a93929-81d6-4f3a-adb2-277a9e004395 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:53:27.009639 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53a93929-81d6-4f3a-adb2-277a9e004395 None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:27.010112 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53a93929-81d6-4f3a-adb2-277a9e004395 None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:27.010615 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53a93929-81d6-4f3a-adb2-277a9e004395 None admin] PATH_INFO: `/v3/users/ff3f9868e06e488a99e9902368f1a78a` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:27.015770 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53a93929-81d6-4f3a-adb2-277a9e004395 None admin] RBAC: Authorizing `identity:get_user(user_id=ff3f9868e06e488a99e9902368f1a78a)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:53:27.017290 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53a93929-81d6-4f3a-adb2-277a9e004395 None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:53:27.019016 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 128/263] 10.222.0.50 () {56 vars in 1177 bytes} [Tue Sep 13 15:53:26 2022] GET /identity/v3/users/ff3f9868e06e488a99e9902368f1a78a => generated 283 bytes in 292 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 13 15:53:27.026804 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53a93929-81d6-4f3a-adb2-277a9e004395 None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:53:27.037897 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 136/264] 10.222.0.50 () {58 vars in 1282 bytes} [Tue Sep 13 15:53:26 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 466 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Sep 13 15:53:27.047965 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66a3fe2b-d6e4-4a82-b9ac-30d05c195309 None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:53:27.196648 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53a93929-81d6-4f3a-adb2-277a9e004395 None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:53:27.237155 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:53:27.249215 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66a3fe2b-d6e4-4a82-b9ac-30d05c195309 None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:53:27.252641 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-969b6311-bd1a-41c2-b6e0-639cd1d49443 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:53:27.253273 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-969b6311-bd1a-41c2-b6e0-639cd1d49443 None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:27.253558 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-969b6311-bd1a-41c2-b6e0-639cd1d49443 None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:27.253834 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-969b6311-bd1a-41c2-b6e0-639cd1d49443 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:27.300383 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4a67eb8-dc74-43de-8bc6-f497444894b5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:53:27.300962 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4a67eb8-dc74-43de-8bc6-f497444894b5 None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:27.301249 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4a67eb8-dc74-43de-8bc6-f497444894b5 None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:27.301531 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4a67eb8-dc74-43de-8bc6-f497444894b5 None admin] PATH_INFO: `/v3/projects/9ec1adca55204df2b653531664b60123` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:27.302691 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4a67eb8-dc74-43de-8bc6-f497444894b5 None admin] RBAC: Authorizing `identity:get_project(project_id=9ec1adca55204df2b653531664b60123)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:53:27.304190 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4a67eb8-dc74-43de-8bc6-f497444894b5 None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:53:27.306147 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 137/265] 10.222.0.50 () {56 vars in 1186 bytes} [Tue Sep 13 15:53:27 2022] GET /identity/v3/projects/9ec1adca55204df2b653531664b60123 => generated 302 bytes in 258 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:53:27.312750 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4a67eb8-dc74-43de-8bc6-f497444894b5 None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:53:27.361452 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-969b6311-bd1a-41c2-b6e0-639cd1d49443 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:53:27.362976 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-969b6311-bd1a-41c2-b6e0-639cd1d49443 None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:53:27.409671 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:53:27.505927 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 129/266] 10.222.0.50 () {58 vars in 1282 bytes} [Tue Sep 13 15:53:27 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 479 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Sep 13 15:53:27.512010 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-969b6311-bd1a-41c2-b6e0-639cd1d49443 None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:53:27.516992 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4a67eb8-dc74-43de-8bc6-f497444894b5 None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:53:27.587635 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b66b14d-83eb-47a9-99bf-9bcb0519e68c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:53:27.588262 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b66b14d-83eb-47a9-99bf-9bcb0519e68c None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:27.588643 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b66b14d-83eb-47a9-99bf-9bcb0519e68c None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:27.589013 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b66b14d-83eb-47a9-99bf-9bcb0519e68c None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:27.589756 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b66b14d-83eb-47a9-99bf-9bcb0519e68c None admin] RBAC: Adding query filter params (role.id=031d4e1a23e34f71858fc4a6dfbd4ccb, scope.project.id=9ec1adca55204df2b653531664b60123, user.id=ff3f9868e06e488a99e9902368f1a78a) {{(pid=81437) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:53:27.590065 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b66b14d-83eb-47a9-99bf-9bcb0519e68c None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:53:27.590555 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:53:27.591565 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b66b14d-83eb-47a9-99bf-9bcb0519e68c None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:53:27.601411 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 138/267] 10.222.0.50 () {56 vars in 1374 bytes} [Tue Sep 13 15:53:27 2022] GET /identity/v3/role_assignments?user.id=ff3f9868e06e488a99e9902368f1a78a&scope.project.id=9ec1adca55204df2b653531664b60123&role.id=031d4e1a23e34f71858fc4a6dfbd4ccb => generated 608 bytes in 289 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:53:27.736713 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-969b6311-bd1a-41c2-b6e0-639cd1d49443 None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:53:27.788035 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd435621-1363-4fc0-b1c8-3549b5fb8318 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:53:27.788696 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd435621-1363-4fc0-b1c8-3549b5fb8318 None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:27.788979 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd435621-1363-4fc0-b1c8-3549b5fb8318 None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:27.789258 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd435621-1363-4fc0-b1c8-3549b5fb8318 None admin] PATH_INFO: `/v3/projects/9ec1adca55204df2b653531664b60123` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:27.794227 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd435621-1363-4fc0-b1c8-3549b5fb8318 None admin] RBAC: Authorizing `identity:get_project(project_id=9ec1adca55204df2b653531664b60123)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:53:27.795762 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd435621-1363-4fc0-b1c8-3549b5fb8318 None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:53:27.797865 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 130/268] 10.222.0.50 () {56 vars in 1186 bytes} [Tue Sep 13 15:53:27 2022] GET /identity/v3/projects/9ec1adca55204df2b653531664b60123 => generated 302 bytes in 286 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:53:27.803469 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b66b14d-83eb-47a9-99bf-9bcb0519e68c None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:53:27.933267 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:53:27.942101 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b66b14d-83eb-47a9-99bf-9bcb0519e68c None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:53:27.989305 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8656bdb-cfce-4a8d-baf6-f1c9e669ee14 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:53:27.989928 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8656bdb-cfce-4a8d-baf6-f1c9e669ee14 None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:27.990542 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8656bdb-cfce-4a8d-baf6-f1c9e669ee14 None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:27.990821 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8656bdb-cfce-4a8d-baf6-f1c9e669ee14 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:27.991586 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8656bdb-cfce-4a8d-baf6-f1c9e669ee14 None admin] RBAC: Adding query filter params (role.id=f74e3f6c20bb426ca030f211c3a9aef5, scope.project.id=9ec1adca55204df2b653531664b60123, user.id=ff3f9868e06e488a99e9902368f1a78a) {{(pid=81437) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:53:27.991898 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8656bdb-cfce-4a8d-baf6-f1c9e669ee14 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:53:27.993355 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8656bdb-cfce-4a8d-baf6-f1c9e669ee14 None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:53:28.000725 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 139/269] 10.222.0.50 () {56 vars in 1374 bytes} [Tue Sep 13 15:53:27 2022] GET /identity/v3/role_assignments?user.id=ff3f9868e06e488a99e9902368f1a78a&scope.project.id=9ec1adca55204df2b653531664b60123&role.id=f74e3f6c20bb426ca030f211c3a9aef5 => generated 608 bytes in 198 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:53:28.024537 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e8fcab0-1402-41d9-8df3-853706c1c0f0 None None] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:28.025180 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e8fcab0-1402-41d9-8df3-853706c1c0f0 None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:28.025651 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e8fcab0-1402-41d9-8df3-853706c1c0f0 None None] PATH_INFO: `/` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:28.028011 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 131/270] 10.222.0.50 () {54 vars in 927 bytes} [Tue Sep 13 15:53:28 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:53:28.034280 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2304d7ca-6f03-47ee-ad69-8f5d5c4e1aa0 None None] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:28.034858 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2304d7ca-6f03-47ee-ad69-8f5d5c4e1aa0 None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:28.035373 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2304d7ca-6f03-47ee-ad69-8f5d5c4e1aa0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:28.055207 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.auth.core [None req-2304d7ca-6f03-47ee-ad69-8f5d5c4e1aa0 None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81437) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:53:28.056799 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-2304d7ca-6f03-47ee-ad69-8f5d5c4e1aa0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:53:28.153081 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 140/271] 10.222.0.50 () {58 vars in 1024 bytes} [Tue Sep 13 15:53:28 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 120 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 13 15:53:28.246083 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee1a93bc-6829-40d1-9078-53eafaa72963 None None] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:28.246568 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee1a93bc-6829-40d1-9078-53eafaa72963 None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:28.246568 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee1a93bc-6829-40d1-9078-53eafaa72963 None None] PATH_INFO: `/` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:28.248179 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 132/272] 10.222.0.50 () {54 vars in 927 bytes} [Tue Sep 13 15:53:28 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:53:28.253397 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56bc9ea7-32c9-4456-9a9a-e713d4e5c612 None None] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:28.253686 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56bc9ea7-32c9-4456-9a9a-e713d4e5c612 None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:28.253961 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56bc9ea7-32c9-4456-9a9a-e713d4e5c612 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:28.275673 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.auth.core [None req-56bc9ea7-32c9-4456-9a9a-e713d4e5c612 None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81437) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:53:28.277711 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-56bc9ea7-32c9-4456-9a9a-e713d4e5c612 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:53:28.372220 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 141/273] 10.222.0.50 () {58 vars in 1024 bytes} [Tue Sep 13 15:53:28 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 120 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 13 15:53:28.458449 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b780540-6bd5-485f-ae5e-7c868938e2a2 None None] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:28.458787 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b780540-6bd5-485f-ae5e-7c868938e2a2 None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:28.459028 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b780540-6bd5-485f-ae5e-7c868938e2a2 None None] PATH_INFO: `/` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:28.460442 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 133/274] 10.222.0.50 () {54 vars in 927 bytes} [Tue Sep 13 15:53:28 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:53:28.466650 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8329927a-77c7-4fc8-8095-c044087e7f8d None None] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:28.466964 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8329927a-77c7-4fc8-8095-c044087e7f8d None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:28.467241 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8329927a-77c7-4fc8-8095-c044087e7f8d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:28.491922 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.auth.core [None req-8329927a-77c7-4fc8-8095-c044087e7f8d None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81437) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:53:28.493114 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-8329927a-77c7-4fc8-8095-c044087e7f8d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:53:28.516467 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:53:28.523226 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b780540-6bd5-485f-ae5e-7c868938e2a2 None None] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:53:28.590465 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 142/275] 10.222.0.50 () {58 vars in 1024 bytes} [Tue Sep 13 15:53:28 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 124 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 13 15:53:28.640635 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:53:28.644595 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b780540-6bd5-485f-ae5e-7c868938e2a2 None None] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:53:28.676030 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8329927a-77c7-4fc8-8095-c044087e7f8d None None] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:53:28.680832 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2636d05-7939-47c5-be9b-c448fec7d685 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:53:28.681468 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2636d05-7939-47c5-be9b-c448fec7d685 None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:28.681751 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2636d05-7939-47c5-be9b-c448fec7d685 None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:28.682026 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2636d05-7939-47c5-be9b-c448fec7d685 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:28.694285 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2636d05-7939-47c5-be9b-c448fec7d685 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 13 15:53:28.700017 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application [None req-d2636d05-7939-47c5-be9b-c448fec7d685 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 13 15:53:28.700017 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:53:28.700017 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:53:28.700017 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:53:28.700017 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:53:28.700017 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:53:28.700017 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:53:28.700017 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:53:28.700017 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:53:28.700017 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:53:28.700017 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:53:28.700017 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:53:28.700017 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 13 15:53:28.700017 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 13 15:53:28.700017 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 13 15:53:28.700017 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 13 15:53:28.702084 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 13 15:53:28.702084 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 13 15:53:28.702084 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 13 15:53:28.702084 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 13 15:53:28.702084 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:53:28.702084 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:53:28.702084 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:53:28.702084 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:53:28.702084 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:53:28.702084 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:53:28.702084 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:53:28.702084 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application with Lock( Sep 13 15:53:28.702084 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:53:28.702084 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:53:28.702084 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:53:28.702084 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:53:28.703674 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:53:28.703674 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:53:28.703674 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:53:28.703674 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:53:28.703674 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 13 15:53:28.703674 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 13 15:53:28.703674 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 13 15:53:28.703674 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 13 15:53:28.703674 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 13 15:53:28.703674 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 13 15:53:28.703674 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 13 15:53:28.703674 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Sep 13 15:53:28.703674 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 134/276] 10.222.0.50 () {56 vars in 1099 bytes} [Tue Sep 13 15:53:28 2022] GET /identity/v3/roles/member => generated 84 bytes in 179 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:53:28.706804 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2636d05-7939-47c5-be9b-c448fec7d685 None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:53:28.820066 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8329927a-77c7-4fc8-8095-c044087e7f8d None None] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:53:28.821830 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2636d05-7939-47c5-be9b-c448fec7d685 None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:53:28.857983 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39f31ae7-ae49-4b25-93ab-a2015f1351c1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:53:28.858199 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-088069a7-162a-4627-b051-5b14e37184af None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:53:28.858719 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-088069a7-162a-4627-b051-5b14e37184af None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:28.859029 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-088069a7-162a-4627-b051-5b14e37184af None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:28.859393 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-088069a7-162a-4627-b051-5b14e37184af None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:28.859933 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39f31ae7-ae49-4b25-93ab-a2015f1351c1 None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:28.860411 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39f31ae7-ae49-4b25-93ab-a2015f1351c1 None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:28.860895 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39f31ae7-ae49-4b25-93ab-a2015f1351c1 None admin] PATH_INFO: `/v3/roles` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:28.861795 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39f31ae7-ae49-4b25-93ab-a2015f1351c1 None admin] RBAC: Adding query filter params (name=member) {{(pid=81438) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:53:28.862306 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39f31ae7-ae49-4b25-93ab-a2015f1351c1 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:53:28.863996 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39f31ae7-ae49-4b25-93ab-a2015f1351c1 None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:53:28.872812 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-088069a7-162a-4627-b051-5b14e37184af None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 13 15:53:28.874317 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 135/277] 10.222.0.50 () {56 vars in 1101 bytes} [Tue Sep 13 15:53:28 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 13 15:53:28.875969 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39f31ae7-ae49-4b25-93ab-a2015f1351c1 None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:53:28.877958 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application [None req-088069a7-162a-4627-b051-5b14e37184af None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Sep 13 15:53:28.877958 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:53:28.877958 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:53:28.877958 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:53:28.877958 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:53:28.877958 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:53:28.877958 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:53:28.877958 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:53:28.877958 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:53:28.877958 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:53:28.877958 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:53:28.877958 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:53:28.877958 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 13 15:53:28.877958 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 13 15:53:28.877958 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 13 15:53:28.877958 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 13 15:53:28.878935 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 13 15:53:28.878935 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 13 15:53:28.878935 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 13 15:53:28.878935 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 13 15:53:28.878935 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:53:28.878935 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:53:28.878935 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:53:28.878935 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:53:28.878935 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:53:28.878935 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:53:28.878935 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:53:28.878935 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application with Lock( Sep 13 15:53:28.878935 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:53:28.878935 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:53:28.878935 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:53:28.878935 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:53:28.879743 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:53:28.879743 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:53:28.879743 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:53:28.879743 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:53:28.879743 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 13 15:53:28.879743 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 13 15:53:28.879743 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 13 15:53:28.879743 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 13 15:53:28.879743 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 13 15:53:28.879743 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 13 15:53:28.879743 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Sep 13 15:53:28.879743 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application Sep 13 15:53:28.883094 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 143/278] 10.222.0.50 () {56 vars in 1099 bytes} [Tue Sep 13 15:53:28 2022] GET /identity/v3/roles/reader => generated 84 bytes in 208 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 13 15:53:28.884506 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-088069a7-162a-4627-b051-5b14e37184af None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:53:28.902666 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:53:28.908242 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:53:28.990783 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39f31ae7-ae49-4b25-93ab-a2015f1351c1 None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:53:28.991097 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-088069a7-162a-4627-b051-5b14e37184af None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:53:29.025512 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d300d63-ce4d-486f-8ee7-053787be9723 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:53:29.026088 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68e9eaa8-b9aa-47b3-90eb-8a99d424dd7a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:53:29.026222 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d300d63-ce4d-486f-8ee7-053787be9723 None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:29.026467 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d300d63-ce4d-486f-8ee7-053787be9723 None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:29.026737 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68e9eaa8-b9aa-47b3-90eb-8a99d424dd7a None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:29.026845 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d300d63-ce4d-486f-8ee7-053787be9723 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:29.027061 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68e9eaa8-b9aa-47b3-90eb-8a99d424dd7a None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:29.027338 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68e9eaa8-b9aa-47b3-90eb-8a99d424dd7a None admin] PATH_INFO: `/v3/users/0878d73c1462483abf59bb21a38302a7` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:29.027864 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d300d63-ce4d-486f-8ee7-053787be9723 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:53:29.029335 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d300d63-ce4d-486f-8ee7-053787be9723 None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:53:29.034965 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 136/279] 10.222.0.50 () {56 vars in 1108 bytes} [Tue Sep 13 15:53:28 2022] GET /identity/v3/domains/default => generated 199 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:53:29.034965 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d300d63-ce4d-486f-8ee7-053787be9723 None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:53:29.040897 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68e9eaa8-b9aa-47b3-90eb-8a99d424dd7a None admin] RBAC: Authorizing `identity:get_user(user_id=0878d73c1462483abf59bb21a38302a7)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:53:29.042449 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68e9eaa8-b9aa-47b3-90eb-8a99d424dd7a None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:53:29.048902 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 144/280] 10.222.0.50 () {56 vars in 1177 bytes} [Tue Sep 13 15:53:28 2022] GET /identity/v3/users/0878d73c1462483abf59bb21a38302a7 => generated 293 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:53:29.050169 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-216e6464-8d1f-461b-8a49-33a0586736f3 None None] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:29.050458 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-216e6464-8d1f-461b-8a49-33a0586736f3 None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:29.050741 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-216e6464-8d1f-461b-8a49-33a0586736f3 None None] PATH_INFO: `/` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:29.051982 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 145/281] 10.222.0.50 () {54 vars in 927 bytes} [Tue Sep 13 15:53:29 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:53:29.056018 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-216e6464-8d1f-461b-8a49-33a0586736f3 None None] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:53:29.123006 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:53:29.124488 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:53:29.133755 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d300d63-ce4d-486f-8ee7-053787be9723 None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:53:29.155203 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-216e6464-8d1f-461b-8a49-33a0586736f3 None None] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:53:29.167508 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f390239-d7ee-4261-8975-fa813ff6eb49 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:53:29.168299 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f390239-d7ee-4261-8975-fa813ff6eb49 None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:29.168578 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f390239-d7ee-4261-8975-fa813ff6eb49 None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:29.168852 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f390239-d7ee-4261-8975-fa813ff6eb49 None admin] PATH_INFO: `/v3/roles` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:29.169667 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f390239-d7ee-4261-8975-fa813ff6eb49 None admin] RBAC: Adding query filter params (name=reader) {{(pid=81438) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:53:29.170335 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f390239-d7ee-4261-8975-fa813ff6eb49 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:53:29.171764 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f390239-d7ee-4261-8975-fa813ff6eb49 None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:53:29.179623 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 137/282] 10.222.0.50 () {56 vars in 1101 bytes} [Tue Sep 13 15:53:29 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:53:29.180407 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f390239-d7ee-4261-8975-fa813ff6eb49 None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:53:29.191040 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ac1cffb-bb43-4ce5-a972-7806046c959b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:53:29.191653 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ac1cffb-bb43-4ce5-a972-7806046c959b None admin] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:29.191933 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ac1cffb-bb43-4ce5-a972-7806046c959b None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:29.192209 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ac1cffb-bb43-4ce5-a972-7806046c959b None admin] PATH_INFO: `/v3/users` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:29.192857 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ac1cffb-bb43-4ce5-a972-7806046c959b None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:53:29.194303 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ac1cffb-bb43-4ce5-a972-7806046c959b None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:53:29.214656 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 146/283] 10.222.0.50 () {60 vars in 1132 bytes} [Tue Sep 13 15:53:29 2022] POST /identity/v3/users => generated 293 bytes in 162 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 15:53:29.215883 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2ceccca-e989-4496-9762-3993b35ed02d None None] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:29.216174 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2ceccca-e989-4496-9762-3993b35ed02d None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:29.216451 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2ceccca-e989-4496-9762-3993b35ed02d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:29.235892 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.auth.core [None req-e2ceccca-e989-4496-9762-3993b35ed02d None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81437) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:53:29.237122 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-e2ceccca-e989-4496-9762-3993b35ed02d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:53:29.310707 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f390239-d7ee-4261-8975-fa813ff6eb49 None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:53:29.318937 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 147/284] 10.222.0.50 () {58 vars in 1024 bytes} [Tue Sep 13 15:53:29 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 13 15:53:29.328567 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2ceccca-e989-4496-9762-3993b35ed02d None None] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:53:29.350484 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40cd578e-f600-4991-a65e-0ebb1fc4cf28 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:53:29.351322 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40cd578e-f600-4991-a65e-0ebb1fc4cf28 None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:29.351704 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40cd578e-f600-4991-a65e-0ebb1fc4cf28 None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:29.352081 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40cd578e-f600-4991-a65e-0ebb1fc4cf28 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:29.353097 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40cd578e-f600-4991-a65e-0ebb1fc4cf28 None admin] RBAC: Adding query filter params (role.id=15db4aaca3264ae29bde38903ba43471, user.id=0878d73c1462483abf59bb21a38302a7, scope.system=all) {{(pid=81438) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:53:29.353321 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40cd578e-f600-4991-a65e-0ebb1fc4cf28 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:53:29.355191 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40cd578e-f600-4991-a65e-0ebb1fc4cf28 None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:53:29.367414 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 138/285] 10.222.0.50 () {56 vars in 1308 bytes} [Tue Sep 13 15:53:29 2022] GET /identity/v3/role_assignments?user.id=0878d73c1462483abf59bb21a38302a7&scope.system=all&role.id=15db4aaca3264ae29bde38903ba43471 => generated 226 bytes in 187 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:53:29.409147 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:53:29.465695 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2ceccca-e989-4496-9762-3993b35ed02d None None] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:53:29.505693 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-111762e0-e9d3-47cf-b73b-5fb4017b9eaf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:53:29.506604 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-111762e0-e9d3-47cf-b73b-5fb4017b9eaf None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:29.507092 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-111762e0-e9d3-47cf-b73b-5fb4017b9eaf None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:29.507511 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-111762e0-e9d3-47cf-b73b-5fb4017b9eaf None admin] PATH_INFO: `/v3/users/4f274b7d64b642e39dd6be3b8cf368df` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:29.508976 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-111762e0-e9d3-47cf-b73b-5fb4017b9eaf None admin] RBAC: Authorizing `identity:get_user(user_id=4f274b7d64b642e39dd6be3b8cf368df)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:53:29.510650 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-111762e0-e9d3-47cf-b73b-5fb4017b9eaf None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:53:29.512573 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 148/286] 10.222.0.50 () {56 vars in 1177 bytes} [Tue Sep 13 15:53:29 2022] GET /identity/v3/users/4f274b7d64b642e39dd6be3b8cf368df => generated 297 bytes in 193 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:53:29.518944 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40cd578e-f600-4991-a65e-0ebb1fc4cf28 None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:53:29.542258 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:53:29.593346 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-111762e0-e9d3-47cf-b73b-5fb4017b9eaf None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:53:29.640097 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40cd578e-f600-4991-a65e-0ebb1fc4cf28 None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:53:29.690346 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-583797aa-c1f1-49c5-b171-bb3d4291e8bb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:53:29.691103 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-583797aa-c1f1-49c5-b171-bb3d4291e8bb None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:29.691613 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-583797aa-c1f1-49c5-b171-bb3d4291e8bb None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:29.691917 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-583797aa-c1f1-49c5-b171-bb3d4291e8bb None admin] PATH_INFO: `/v3/projects/9ec1adca55204df2b653531664b60123` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:29.693388 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-583797aa-c1f1-49c5-b171-bb3d4291e8bb None admin] RBAC: Authorizing `identity:get_project(project_id=9ec1adca55204df2b653531664b60123)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:53:29.695026 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-583797aa-c1f1-49c5-b171-bb3d4291e8bb None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:53:29.697217 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 139/287] 10.222.0.50 () {56 vars in 1186 bytes} [Tue Sep 13 15:53:29 2022] GET /identity/v3/projects/9ec1adca55204df2b653531664b60123 => generated 302 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 13 15:53:29.703732 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-583797aa-c1f1-49c5-b171-bb3d4291e8bb None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:53:29.741375 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-111762e0-e9d3-47cf-b73b-5fb4017b9eaf None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:53:29.780019 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:53:29.794033 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a6621a2-cd66-4049-8b6a-f2b9faea8e9f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:53:29.795371 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a6621a2-cd66-4049-8b6a-f2b9faea8e9f None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:29.795837 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a6621a2-cd66-4049-8b6a-f2b9faea8e9f None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:29.796322 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a6621a2-cd66-4049-8b6a-f2b9faea8e9f None admin] PATH_INFO: `/v3/roles/member` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:29.804003 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:53:29.812939 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a6621a2-cd66-4049-8b6a-f2b9faea8e9f None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 13 15:53:29.821346 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application [None req-9a6621a2-cd66-4049-8b6a-f2b9faea8e9f None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 13 15:53:29.821346 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:53:29.821346 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:53:29.821346 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:53:29.821346 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:53:29.821346 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:53:29.821346 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:53:29.821346 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:53:29.821346 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:53:29.821346 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:53:29.821346 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:53:29.821346 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:53:29.821346 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 13 15:53:29.821346 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 13 15:53:29.821346 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 13 15:53:29.821346 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 13 15:53:29.822432 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 13 15:53:29.822432 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 13 15:53:29.822432 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 13 15:53:29.822432 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 13 15:53:29.822432 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:53:29.822432 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:53:29.822432 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:53:29.822432 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:53:29.822432 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:53:29.822432 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:53:29.822432 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:53:29.822432 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application with Lock( Sep 13 15:53:29.822432 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:53:29.822432 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:53:29.822432 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:53:29.822432 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:53:29.824046 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:53:29.824046 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:53:29.824046 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:53:29.824046 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:53:29.824046 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 13 15:53:29.824046 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 13 15:53:29.824046 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 13 15:53:29.824046 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 13 15:53:29.824046 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 13 15:53:29.824046 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 13 15:53:29.824046 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 13 15:53:29.824046 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application Sep 13 15:53:29.826268 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 149/288] 10.222.0.50 () {56 vars in 1099 bytes} [Tue Sep 13 15:53:29 2022] GET /identity/v3/roles/member => generated 84 bytes in 233 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:53:29.831102 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-583797aa-c1f1-49c5-b171-bb3d4291e8bb None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:53:29.836004 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a6621a2-cd66-4049-8b6a-f2b9faea8e9f None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:53:29.872361 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-259f5336-f533-4c46-98b2-69d5cae5f9e8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:53:29.874298 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-259f5336-f533-4c46-98b2-69d5cae5f9e8 None admin] REQUEST_METHOD: `PUT` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:29.874785 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-259f5336-f533-4c46-98b2-69d5cae5f9e8 None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:29.878093 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-259f5336-f533-4c46-98b2-69d5cae5f9e8 None admin] PATH_INFO: `/v3/projects/9ec1adca55204df2b653531664b60123/users/4f274b7d64b642e39dd6be3b8cf368df/roles/b2d1670446a54085ad2bdb935f003d25` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:29.881381 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-259f5336-f533-4c46-98b2-69d5cae5f9e8 None admin] RBAC: Authorizing `identity:create_grant(project_id=9ec1adca55204df2b653531664b60123, user_id=4f274b7d64b642e39dd6be3b8cf368df, role_id=b2d1670446a54085ad2bdb935f003d25)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:53:29.883361 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-259f5336-f533-4c46-98b2-69d5cae5f9e8 None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:53:29.909249 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 140/289] 10.222.0.50 () {58 vars in 1439 bytes} [Tue Sep 13 15:53:29 2022] PUT /identity/v3/projects/9ec1adca55204df2b653531664b60123/users/4f274b7d64b642e39dd6be3b8cf368df/roles/b2d1670446a54085ad2bdb935f003d25 => generated 0 bytes in 206 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:53:30.007735 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a6621a2-cd66-4049-8b6a-f2b9faea8e9f None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:53:30.064807 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcfb773e-4b12-4d22-9780-8684e7a36fdb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:53:30.064965 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcfb773e-4b12-4d22-9780-8684e7a36fdb None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:30.065531 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcfb773e-4b12-4d22-9780-8684e7a36fdb None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:30.065531 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcfb773e-4b12-4d22-9780-8684e7a36fdb None admin] PATH_INFO: `/v3/roles` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:30.066294 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dcfb773e-4b12-4d22-9780-8684e7a36fdb None admin] RBAC: Adding query filter params (name=member) {{(pid=81437) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:53:30.066596 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dcfb773e-4b12-4d22-9780-8684e7a36fdb None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:53:30.068023 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dcfb773e-4b12-4d22-9780-8684e7a36fdb None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:53:30.082431 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 150/290] 10.222.0.50 () {56 vars in 1101 bytes} [Tue Sep 13 15:53:29 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 247 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:53:30.088269 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-259f5336-f533-4c46-98b2-69d5cae5f9e8 None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:53:30.230757 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-259f5336-f533-4c46-98b2-69d5cae5f9e8 None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:53:30.267925 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e5713b3-c119-4ca7-a54f-b8243276a814 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:53:30.268488 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e5713b3-c119-4ca7-a54f-b8243276a814 None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:30.268766 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e5713b3-c119-4ca7-a54f-b8243276a814 None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:30.269041 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e5713b3-c119-4ca7-a54f-b8243276a814 None admin] PATH_INFO: `/v3/users/b8269ec40ba14c51825ec2be9c71b254` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:30.270222 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e5713b3-c119-4ca7-a54f-b8243276a814 None admin] RBAC: Authorizing `identity:get_user(user_id=b8269ec40ba14c51825ec2be9c71b254)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:53:30.271712 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e5713b3-c119-4ca7-a54f-b8243276a814 None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:53:30.273654 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 141/291] 10.222.0.50 () {56 vars in 1177 bytes} [Tue Sep 13 15:53:30 2022] GET /identity/v3/users/b8269ec40ba14c51825ec2be9c71b254 => generated 297 bytes in 186 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:53:30.280553 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcfb773e-4b12-4d22-9780-8684e7a36fdb None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:53:30.367897 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:53:30.442301 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcfb773e-4b12-4d22-9780-8684e7a36fdb None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:53:30.491635 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f805e114-b524-4ecd-9a51-8ffeb77508e0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:53:30.492247 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f805e114-b524-4ecd-9a51-8ffeb77508e0 None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:30.492521 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f805e114-b524-4ecd-9a51-8ffeb77508e0 None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:30.492789 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f805e114-b524-4ecd-9a51-8ffeb77508e0 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:30.572548 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f805e114-b524-4ecd-9a51-8ffeb77508e0 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:53:30.574669 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f805e114-b524-4ecd-9a51-8ffeb77508e0 None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:53:30.621550 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:53:30.680316 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 151/292] 10.222.0.50 () {58 vars in 1282 bytes} [Tue Sep 13 15:53:30 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 400 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 13 15:53:30.686692 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e5713b3-c119-4ca7-a54f-b8243276a814 None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:53:30.703519 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:53:30.825772 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e5713b3-c119-4ca7-a54f-b8243276a814 None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:53:30.869574 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2139f5c-3b4b-4cc5-96e1-111ff8490b6a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:53:30.871004 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2139f5c-3b4b-4cc5-96e1-111ff8490b6a None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:30.871292 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2139f5c-3b4b-4cc5-96e1-111ff8490b6a None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:30.871571 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2139f5c-3b4b-4cc5-96e1-111ff8490b6a None admin] PATH_INFO: `/v3/projects/9ec1adca55204df2b653531664b60123` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:30.872723 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2139f5c-3b4b-4cc5-96e1-111ff8490b6a None admin] RBAC: Authorizing `identity:get_project(project_id=9ec1adca55204df2b653531664b60123)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:53:30.874166 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2139f5c-3b4b-4cc5-96e1-111ff8490b6a None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:53:30.875884 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 142/293] 10.222.0.50 () {56 vars in 1186 bytes} [Tue Sep 13 15:53:30 2022] GET /identity/v3/projects/9ec1adca55204df2b653531664b60123 => generated 302 bytes in 190 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:53:30.881751 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f805e114-b524-4ecd-9a51-8ffeb77508e0 None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:53:31.035937 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f805e114-b524-4ecd-9a51-8ffeb77508e0 None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:53:31.072018 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31e795fa-d5f7-46cd-8e19-951c34e2256c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:53:31.072637 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31e795fa-d5f7-46cd-8e19-951c34e2256c None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:31.072975 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31e795fa-d5f7-46cd-8e19-951c34e2256c None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:31.073766 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31e795fa-d5f7-46cd-8e19-951c34e2256c None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:31.074492 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31e795fa-d5f7-46cd-8e19-951c34e2256c None admin] RBAC: Adding query filter params (role.id=15db4aaca3264ae29bde38903ba43471, scope.project.id=9ec1adca55204df2b653531664b60123, user.id=b8269ec40ba14c51825ec2be9c71b254) {{(pid=81437) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:53:31.074806 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31e795fa-d5f7-46cd-8e19-951c34e2256c None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:53:31.076356 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31e795fa-d5f7-46cd-8e19-951c34e2256c None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:53:31.082815 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 152/294] 10.222.0.50 () {56 vars in 1374 bytes} [Tue Sep 13 15:53:30 2022] GET /identity/v3/role_assignments?user.id=b8269ec40ba14c51825ec2be9c71b254&scope.project.id=9ec1adca55204df2b653531664b60123&role.id=15db4aaca3264ae29bde38903ba43471 => generated 608 bytes in 201 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:53:31.898550 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c906f23-3c48-4f18-a530-b3b874ac735f None None] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:31.899124 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c906f23-3c48-4f18-a530-b3b874ac735f None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:31.899124 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c906f23-3c48-4f18-a530-b3b874ac735f None None] PATH_INFO: `/` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:31.900705 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 143/295] 10.222.0.50 () {54 vars in 927 bytes} [Tue Sep 13 15:53:31 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:53:31.906251 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-933858ff-040a-420a-8755-e6084be225a9 None None] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:31.906545 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-933858ff-040a-420a-8755-e6084be225a9 None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:31.906821 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-933858ff-040a-420a-8755-e6084be225a9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:31.925479 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.auth.core [None req-933858ff-040a-420a-8755-e6084be225a9 None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81437) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:53:31.926669 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-933858ff-040a-420a-8755-e6084be225a9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:53:31.935298 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:53:32.003206 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 153/296] 10.222.0.50 () {58 vars in 1024 bytes} [Tue Sep 13 15:53:31 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 13 15:53:32.131893 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1eb1a8e4-9aeb-4814-ac1d-2ff17bf3babb None None] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:32.132266 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1eb1a8e4-9aeb-4814-ac1d-2ff17bf3babb None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:32.132529 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1eb1a8e4-9aeb-4814-ac1d-2ff17bf3babb None None] PATH_INFO: `/` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:32.134031 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 144/297] 10.222.0.50 () {54 vars in 927 bytes} [Tue Sep 13 15:53:32 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:53:32.134650 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e0a224a-a926-4617-93d3-335ff4bbe03f None None] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:32.136070 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e0a224a-a926-4617-93d3-335ff4bbe03f None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:32.136546 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e0a224a-a926-4617-93d3-335ff4bbe03f None None] PATH_INFO: `/` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:32.138419 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 154/298] 10.222.0.50 () {54 vars in 927 bytes} [Tue Sep 13 15:53:32 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:53:32.141889 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e216568-fd75-4a96-8ab3-51eaaef8b407 None None] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:32.142180 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e216568-fd75-4a96-8ab3-51eaaef8b407 None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:32.142461 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e216568-fd75-4a96-8ab3-51eaaef8b407 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:32.143756 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-353b1091-8689-45e6-8189-4ac634860e4b None None] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:32.144239 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-353b1091-8689-45e6-8189-4ac634860e4b None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:32.144750 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-353b1091-8689-45e6-8189-4ac634860e4b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:32.163364 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.auth.core [None req-5e216568-fd75-4a96-8ab3-51eaaef8b407 None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81438) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:53:32.164772 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-5e216568-fd75-4a96-8ab3-51eaaef8b407 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:53:32.169068 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.auth.core [None req-353b1091-8689-45e6-8189-4ac634860e4b None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81437) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:53:32.170253 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-353b1091-8689-45e6-8189-4ac634860e4b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:53:32.248526 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 145/299] 10.222.0.50 () {58 vars in 1024 bytes} [Tue Sep 13 15:53:32 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 13 15:53:32.259768 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 155/300] 10.222.0.50 () {58 vars in 1024 bytes} [Tue Sep 13 15:53:32 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 117 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 13 15:53:32.292222 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e216568-fd75-4a96-8ab3-51eaaef8b407 None None] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:53:32.305223 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:53:32.402219 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e216568-fd75-4a96-8ab3-51eaaef8b407 None None] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:53:32.438222 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e943decd-e866-4add-8430-664bf0981654 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:53:32.438782 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e943decd-e866-4add-8430-664bf0981654 None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:32.439098 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e943decd-e866-4add-8430-664bf0981654 None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:32.439373 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e943decd-e866-4add-8430-664bf0981654 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:32.451026 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e943decd-e866-4add-8430-664bf0981654 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 13 15:53:32.456000 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application [None req-e943decd-e866-4add-8430-664bf0981654 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 13 15:53:32.456000 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:53:32.456000 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:53:32.456000 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:53:32.456000 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:53:32.456000 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:53:32.456000 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:53:32.456000 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:53:32.456000 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:53:32.456000 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:53:32.456000 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:53:32.456000 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:53:32.456000 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 13 15:53:32.456000 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 13 15:53:32.456000 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 13 15:53:32.456000 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 13 15:53:32.456871 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 13 15:53:32.456871 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 13 15:53:32.456871 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 13 15:53:32.456871 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 13 15:53:32.456871 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:53:32.456871 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:53:32.456871 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:53:32.456871 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:53:32.456871 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:53:32.456871 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:53:32.456871 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:53:32.456871 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application with Lock( Sep 13 15:53:32.456871 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:53:32.456871 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:53:32.456871 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:53:32.456871 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:53:32.457659 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:53:32.457659 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:53:32.457659 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:53:32.457659 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:53:32.457659 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 13 15:53:32.457659 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 13 15:53:32.457659 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 13 15:53:32.457659 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 13 15:53:32.457659 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 13 15:53:32.457659 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 13 15:53:32.457659 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 13 15:53:32.457659 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Sep 13 15:53:32.457659 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 146/301] 10.222.0.50 () {56 vars in 1099 bytes} [Tue Sep 13 15:53:32 2022] GET /identity/v3/roles/member => generated 84 bytes in 166 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:53:32.465183 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-353b1091-8689-45e6-8189-4ac634860e4b None None] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:53:32.523555 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e943decd-e866-4add-8430-664bf0981654 None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:53:32.552450 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:53:32.584854 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-353b1091-8689-45e6-8189-4ac634860e4b None None] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:53:32.602820 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:53:32.622759 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b99acb5a-c5a9-42f9-a8a2-38cc22202c83 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:53:32.623400 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b99acb5a-c5a9-42f9-a8a2-38cc22202c83 None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:32.623685 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b99acb5a-c5a9-42f9-a8a2-38cc22202c83 None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:32.623960 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b99acb5a-c5a9-42f9-a8a2-38cc22202c83 None admin] PATH_INFO: `/v3/roles` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:32.624619 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b99acb5a-c5a9-42f9-a8a2-38cc22202c83 None admin] RBAC: Adding query filter params (name=member) {{(pid=81437) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:53:32.624928 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b99acb5a-c5a9-42f9-a8a2-38cc22202c83 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:53:32.626303 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b99acb5a-c5a9-42f9-a8a2-38cc22202c83 None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:53:32.635296 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 156/302] 10.222.0.50 () {56 vars in 1101 bytes} [Tue Sep 13 15:53:32 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:53:32.639270 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e943decd-e866-4add-8430-664bf0981654 None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:53:32.640526 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b99acb5a-c5a9-42f9-a8a2-38cc22202c83 None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:53:32.674093 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15baed9b-8c16-4ad1-86d5-409268b2588a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:53:32.674720 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15baed9b-8c16-4ad1-86d5-409268b2588a None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:32.675051 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15baed9b-8c16-4ad1-86d5-409268b2588a None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:32.675326 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15baed9b-8c16-4ad1-86d5-409268b2588a None admin] PATH_INFO: `/v3/domains/default` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:32.676425 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15baed9b-8c16-4ad1-86d5-409268b2588a None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:53:32.677864 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15baed9b-8c16-4ad1-86d5-409268b2588a None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:53:32.679627 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 147/303] 10.222.0.50 () {56 vars in 1108 bytes} [Tue Sep 13 15:53:32 2022] GET /identity/v3/domains/default => generated 199 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:53:32.680727 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-373f95c5-ed95-40d0-bd77-6c9892241501 None None] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:32.681010 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-373f95c5-ed95-40d0-bd77-6c9892241501 None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:32.681287 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-373f95c5-ed95-40d0-bd77-6c9892241501 None None] PATH_INFO: `/` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:32.682938 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 148/304] 10.222.0.50 () {54 vars in 927 bytes} [Tue Sep 13 15:53:32 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:53:32.687710 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-373f95c5-ed95-40d0-bd77-6c9892241501 None None] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:53:32.741524 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b99acb5a-c5a9-42f9-a8a2-38cc22202c83 None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:53:32.774519 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffc8b1f3-8791-4cf5-bac8-c6a9f1db1f71 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:53:32.775167 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffc8b1f3-8791-4cf5-bac8-c6a9f1db1f71 None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:32.775441 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffc8b1f3-8791-4cf5-bac8-c6a9f1db1f71 None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:32.775711 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffc8b1f3-8791-4cf5-bac8-c6a9f1db1f71 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:32.778375 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:53:32.787458 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffc8b1f3-8791-4cf5-bac8-c6a9f1db1f71 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 13 15:53:32.787585 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-373f95c5-ed95-40d0-bd77-6c9892241501 None None] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:53:32.793003 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application [None req-ffc8b1f3-8791-4cf5-bac8-c6a9f1db1f71 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Sep 13 15:53:32.793003 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:53:32.793003 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:53:32.793003 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:53:32.793003 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:53:32.793003 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:53:32.793003 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:53:32.793003 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:53:32.793003 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:53:32.793003 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:53:32.793003 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:53:32.793003 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:53:32.793003 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 13 15:53:32.793003 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 13 15:53:32.793003 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 13 15:53:32.793003 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 13 15:53:32.794071 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 13 15:53:32.794071 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 13 15:53:32.794071 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 13 15:53:32.794071 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 13 15:53:32.794071 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:53:32.794071 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:53:32.794071 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:53:32.794071 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:53:32.794071 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:53:32.794071 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:53:32.794071 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:53:32.794071 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application with Lock( Sep 13 15:53:32.794071 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:53:32.794071 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:53:32.794071 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:53:32.794071 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:53:32.794940 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:53:32.794940 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:53:32.794940 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:53:32.794940 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:53:32.794940 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 13 15:53:32.794940 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 13 15:53:32.794940 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 13 15:53:32.794940 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 13 15:53:32.794940 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 13 15:53:32.794940 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 13 15:53:32.794940 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Sep 13 15:53:32.794940 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application Sep 13 15:53:32.795878 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 157/305] 10.222.0.50 () {56 vars in 1099 bytes} [Tue Sep 13 15:53:32 2022] GET /identity/v3/roles/reader => generated 84 bytes in 155 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:53:32.795943 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffc8b1f3-8791-4cf5-bac8-c6a9f1db1f71 None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:53:32.824914 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3aca5f53-0091-46ad-a668-7862d0bc7145 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:53:32.825538 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3aca5f53-0091-46ad-a668-7862d0bc7145 None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:32.825815 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3aca5f53-0091-46ad-a668-7862d0bc7145 None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:32.826083 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3aca5f53-0091-46ad-a668-7862d0bc7145 None admin] PATH_INFO: `/v3/users/0878d73c1462483abf59bb21a38302a7` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:32.827348 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3aca5f53-0091-46ad-a668-7862d0bc7145 None admin] RBAC: Authorizing `identity:get_user(user_id=0878d73c1462483abf59bb21a38302a7)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:53:32.829065 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3aca5f53-0091-46ad-a668-7862d0bc7145 None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:53:32.830748 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 149/306] 10.222.0.50 () {56 vars in 1177 bytes} [Tue Sep 13 15:53:32 2022] GET /identity/v3/users/0878d73c1462483abf59bb21a38302a7 => generated 293 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:53:32.831707 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4937ef5e-1b0f-4d29-abf0-a280e02e879b None None] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:32.831984 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4937ef5e-1b0f-4d29-abf0-a280e02e879b None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:32.832252 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4937ef5e-1b0f-4d29-abf0-a280e02e879b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:32.844146 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:53:32.850747 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.auth.core [None req-4937ef5e-1b0f-4d29-abf0-a280e02e879b None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81438) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:53:32.851946 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-4937ef5e-1b0f-4d29-abf0-a280e02e879b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:53:32.897105 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffc8b1f3-8791-4cf5-bac8-c6a9f1db1f71 None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:53:32.919743 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 150/307] 10.222.0.50 () {58 vars in 1024 bytes} [Tue Sep 13 15:53:32 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 13 15:53:32.920728 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4937ef5e-1b0f-4d29-abf0-a280e02e879b None None] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:53:32.934227 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebf990f2-040b-4fab-a65f-9b2c62db2ffc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:53:32.935222 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebf990f2-040b-4fab-a65f-9b2c62db2ffc None admin] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:32.936018 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebf990f2-040b-4fab-a65f-9b2c62db2ffc None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:32.936563 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebf990f2-040b-4fab-a65f-9b2c62db2ffc None admin] PATH_INFO: `/v3/groups` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:32.937426 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebf990f2-040b-4fab-a65f-9b2c62db2ffc None admin] RBAC: Authorizing `identity:create_group()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:53:32.939495 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebf990f2-040b-4fab-a65f-9b2c62db2ffc None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:53:32.959682 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 158/308] 10.222.0.50 () {60 vars in 1134 bytes} [Tue Sep 13 15:53:32 2022] POST /identity/v3/groups => generated 229 bytes in 164 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 15:53:32.961188 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebf990f2-040b-4fab-a65f-9b2c62db2ffc None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:53:33.041687 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:53:33.046600 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4937ef5e-1b0f-4d29-abf0-a280e02e879b None None] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:53:33.096402 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be1a7d48-74b4-40ac-ab08-fab25056bad6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:53:33.097423 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be1a7d48-74b4-40ac-ab08-fab25056bad6 None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:33.097900 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be1a7d48-74b4-40ac-ab08-fab25056bad6 None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:33.098367 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be1a7d48-74b4-40ac-ab08-fab25056bad6 None admin] PATH_INFO: `/v3/roles` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:33.099412 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be1a7d48-74b4-40ac-ab08-fab25056bad6 None admin] RBAC: Adding query filter params (name=reader) {{(pid=81438) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:53:33.099963 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be1a7d48-74b4-40ac-ab08-fab25056bad6 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:53:33.101558 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be1a7d48-74b4-40ac-ab08-fab25056bad6 None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:53:33.106407 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:53:33.113097 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebf990f2-040b-4fab-a65f-9b2c62db2ffc None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:53:33.115436 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 151/309] 10.222.0.50 () {56 vars in 1101 bytes} [Tue Sep 13 15:53:32 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 195 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:53:33.126993 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be1a7d48-74b4-40ac-ab08-fab25056bad6 None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:53:33.159921 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0856035f-b5f1-4e87-89fb-b44f6748b4e8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:53:33.160589 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0856035f-b5f1-4e87-89fb-b44f6748b4e8 None admin] REQUEST_METHOD: `PUT` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:33.160869 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0856035f-b5f1-4e87-89fb-b44f6748b4e8 None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:33.161190 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0856035f-b5f1-4e87-89fb-b44f6748b4e8 None admin] PATH_INFO: `/v3/system/users/0878d73c1462483abf59bb21a38302a7/roles/15db4aaca3264ae29bde38903ba43471` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:33.162569 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0856035f-b5f1-4e87-89fb-b44f6748b4e8 None admin] RBAC: Authorizing `identity:create_system_grant_for_user(user_id=0878d73c1462483abf59bb21a38302a7, role_id=15db4aaca3264ae29bde38903ba43471)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:53:33.164015 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0856035f-b5f1-4e87-89fb-b44f6748b4e8 None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:53:33.178266 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 159/310] 10.222.0.50 () {58 vars in 1334 bytes} [Tue Sep 13 15:53:32 2022] PUT /identity/v3/system/users/0878d73c1462483abf59bb21a38302a7/roles/15db4aaca3264ae29bde38903ba43471 => generated 0 bytes in 217 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:53:33.221540 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0856035f-b5f1-4e87-89fb-b44f6748b4e8 None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:53:33.262108 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be1a7d48-74b4-40ac-ab08-fab25056bad6 None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:53:33.310893 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98eda0f3-2c47-45a7-832f-97c9fb9361db None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:53:33.311601 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98eda0f3-2c47-45a7-832f-97c9fb9361db None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:33.311941 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98eda0f3-2c47-45a7-832f-97c9fb9361db None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:33.312227 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98eda0f3-2c47-45a7-832f-97c9fb9361db None admin] PATH_INFO: `/v3/users/8f5df6f7cab84a5bb067824eb0b7c534` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:33.332341 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98eda0f3-2c47-45a7-832f-97c9fb9361db None admin] RBAC: Authorizing `identity:get_user(user_id=8f5df6f7cab84a5bb067824eb0b7c534)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:53:33.333962 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98eda0f3-2c47-45a7-832f-97c9fb9361db None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:53:33.336146 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 152/311] 10.222.0.50 () {56 vars in 1177 bytes} [Tue Sep 13 15:53:33 2022] GET /identity/v3/users/8f5df6f7cab84a5bb067824eb0b7c534 => generated 293 bytes in 214 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:53:33.341811 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98eda0f3-2c47-45a7-832f-97c9fb9361db None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:53:33.362942 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0856035f-b5f1-4e87-89fb-b44f6748b4e8 None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:53:33.386310 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:53:33.410322 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9eb66568-eb86-43a5-b263-70e2d3233906 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:53:33.411275 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9eb66568-eb86-43a5-b263-70e2d3233906 None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:33.411556 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9eb66568-eb86-43a5-b263-70e2d3233906 None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:33.411831 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9eb66568-eb86-43a5-b263-70e2d3233906 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:33.414479 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:53:33.431434 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9eb66568-eb86-43a5-b263-70e2d3233906 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 13 15:53:33.437786 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application [None req-9eb66568-eb86-43a5-b263-70e2d3233906 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Sep 13 15:53:33.437786 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:53:33.437786 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:53:33.437786 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:53:33.437786 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:53:33.437786 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:53:33.437786 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:53:33.437786 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:53:33.437786 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:53:33.437786 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:53:33.437786 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:53:33.437786 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:53:33.437786 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 13 15:53:33.437786 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 13 15:53:33.437786 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 13 15:53:33.437786 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 13 15:53:33.438979 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 13 15:53:33.438979 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 13 15:53:33.438979 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 13 15:53:33.438979 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 13 15:53:33.438979 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:53:33.438979 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:53:33.438979 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:53:33.438979 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:53:33.438979 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:53:33.438979 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:53:33.438979 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:53:33.438979 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application with Lock( Sep 13 15:53:33.438979 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:53:33.438979 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:53:33.438979 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:53:33.438979 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:53:33.440055 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:53:33.440055 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:53:33.440055 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:53:33.440055 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:53:33.440055 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 13 15:53:33.440055 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 13 15:53:33.440055 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 13 15:53:33.440055 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 13 15:53:33.440055 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 13 15:53:33.440055 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 13 15:53:33.440055 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Sep 13 15:53:33.440055 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application Sep 13 15:53:33.440055 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 160/312] 10.222.0.50 () {56 vars in 1099 bytes} [Tue Sep 13 15:53:33 2022] GET /identity/v3/roles/reader => generated 84 bytes in 217 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 13 15:53:33.444890 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9eb66568-eb86-43a5-b263-70e2d3233906 None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:53:33.475163 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98eda0f3-2c47-45a7-832f-97c9fb9361db None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:53:33.515844 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5bf7e6b-8e0a-4a95-8512-b59d27e5b885 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:53:33.516862 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5bf7e6b-8e0a-4a95-8512-b59d27e5b885 None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:33.517368 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5bf7e6b-8e0a-4a95-8512-b59d27e5b885 None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:33.517839 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5bf7e6b-8e0a-4a95-8512-b59d27e5b885 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:33.518755 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5bf7e6b-8e0a-4a95-8512-b59d27e5b885 None admin] RBAC: Adding query filter params (role.id=b2d1670446a54085ad2bdb935f003d25, user.id=8f5df6f7cab84a5bb067824eb0b7c534, scope.system=all) {{(pid=81438) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:53:33.519282 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5bf7e6b-8e0a-4a95-8512-b59d27e5b885 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:53:33.521428 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5bf7e6b-8e0a-4a95-8512-b59d27e5b885 None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:53:33.528787 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 153/313] 10.222.0.50 () {56 vars in 1308 bytes} [Tue Sep 13 15:53:33 2022] GET /identity/v3/role_assignments?user.id=8f5df6f7cab84a5bb067824eb0b7c534&scope.system=all&role.id=b2d1670446a54085ad2bdb935f003d25 => generated 226 bytes in 187 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 13 15:53:33.580148 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9eb66568-eb86-43a5-b263-70e2d3233906 None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:53:33.620554 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cca9cc75-3328-45ba-8d29-7f02d88458a5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:53:33.621182 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cca9cc75-3328-45ba-8d29-7f02d88458a5 None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:33.621459 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cca9cc75-3328-45ba-8d29-7f02d88458a5 None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:33.621730 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cca9cc75-3328-45ba-8d29-7f02d88458a5 None admin] PATH_INFO: `/v3/roles` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:33.622419 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cca9cc75-3328-45ba-8d29-7f02d88458a5 None admin] RBAC: Adding query filter params (name=reader) {{(pid=81437) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:53:33.622723 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cca9cc75-3328-45ba-8d29-7f02d88458a5 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:53:33.624376 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cca9cc75-3328-45ba-8d29-7f02d88458a5 None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:53:33.634607 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 161/314] 10.222.0.50 () {56 vars in 1101 bytes} [Tue Sep 13 15:53:33 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 190 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 13 15:53:33.640792 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5bf7e6b-8e0a-4a95-8512-b59d27e5b885 None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:53:33.761209 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5bf7e6b-8e0a-4a95-8512-b59d27e5b885 None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:53:33.783942 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:53:33.799729 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23182ecb-af9e-46b4-bf5c-bc13961ee526 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:53:33.800378 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23182ecb-af9e-46b4-bf5c-bc13961ee526 None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:33.800663 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23182ecb-af9e-46b4-bf5c-bc13961ee526 None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:33.800939 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23182ecb-af9e-46b4-bf5c-bc13961ee526 None admin] PATH_INFO: `/v3/users/4f274b7d64b642e39dd6be3b8cf368df` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:33.802079 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23182ecb-af9e-46b4-bf5c-bc13961ee526 None admin] RBAC: Authorizing `identity:get_user(user_id=4f274b7d64b642e39dd6be3b8cf368df)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:53:33.803683 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23182ecb-af9e-46b4-bf5c-bc13961ee526 None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:53:33.805606 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 154/315] 10.222.0.50 () {56 vars in 1177 bytes} [Tue Sep 13 15:53:33 2022] GET /identity/v3/users/4f274b7d64b642e39dd6be3b8cf368df => generated 297 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:53:33.811795 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cca9cc75-3328-45ba-8d29-7f02d88458a5 None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:53:33.856145 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:53:33.920063 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cca9cc75-3328-45ba-8d29-7f02d88458a5 None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:53:33.961732 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83b79b8e-768f-47ae-971e-af5e09559e9d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:53:33.962352 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83b79b8e-768f-47ae-971e-af5e09559e9d None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:33.962632 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83b79b8e-768f-47ae-971e-af5e09559e9d None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:33.962925 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83b79b8e-768f-47ae-971e-af5e09559e9d None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:34.036360 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83b79b8e-768f-47ae-971e-af5e09559e9d None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:53:34.037918 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83b79b8e-768f-47ae-971e-af5e09559e9d None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:53:34.081462 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:53:34.143861 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 162/316] 10.222.0.50 () {58 vars in 1282 bytes} [Tue Sep 13 15:53:33 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 332 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 13 15:53:34.149429 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23182ecb-af9e-46b4-bf5c-bc13961ee526 None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:53:34.260025 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23182ecb-af9e-46b4-bf5c-bc13961ee526 None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:53:34.301128 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2c423f6-4aa8-4a67-9728-d1272e58f3d6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:53:34.301766 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2c423f6-4aa8-4a67-9728-d1272e58f3d6 None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:34.302045 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2c423f6-4aa8-4a67-9728-d1272e58f3d6 None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:34.302319 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2c423f6-4aa8-4a67-9728-d1272e58f3d6 None admin] PATH_INFO: `/v3/projects/9ec1adca55204df2b653531664b60123` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:34.303459 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2c423f6-4aa8-4a67-9728-d1272e58f3d6 None admin] RBAC: Authorizing `identity:get_project(project_id=9ec1adca55204df2b653531664b60123)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:53:34.304997 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2c423f6-4aa8-4a67-9728-d1272e58f3d6 None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:53:34.307015 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 155/317] 10.222.0.50 () {56 vars in 1186 bytes} [Tue Sep 13 15:53:34 2022] GET /identity/v3/projects/9ec1adca55204df2b653531664b60123 => generated 302 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:53:34.316040 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83b79b8e-768f-47ae-971e-af5e09559e9d None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:53:34.420919 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83b79b8e-768f-47ae-971e-af5e09559e9d None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:53:34.458601 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96d81c1c-47db-43e1-8138-62d5896231fd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:53:34.459243 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96d81c1c-47db-43e1-8138-62d5896231fd None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:34.459521 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96d81c1c-47db-43e1-8138-62d5896231fd None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:34.459795 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96d81c1c-47db-43e1-8138-62d5896231fd None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:34.460532 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96d81c1c-47db-43e1-8138-62d5896231fd None admin] RBAC: Adding query filter params (role.id=b2d1670446a54085ad2bdb935f003d25, scope.project.id=9ec1adca55204df2b653531664b60123, user.id=4f274b7d64b642e39dd6be3b8cf368df) {{(pid=81437) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:53:34.460840 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96d81c1c-47db-43e1-8138-62d5896231fd None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:53:34.462269 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96d81c1c-47db-43e1-8138-62d5896231fd None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:53:34.469467 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 163/318] 10.222.0.50 () {56 vars in 1374 bytes} [Tue Sep 13 15:53:34 2022] GET /identity/v3/role_assignments?user.id=4f274b7d64b642e39dd6be3b8cf368df&scope.project.id=9ec1adca55204df2b653531664b60123&role.id=b2d1670446a54085ad2bdb935f003d25 => generated 608 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:53:35.243080 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b09a1919-a09d-4c36-96aa-8fab2b889709 None None] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:35.243583 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b09a1919-a09d-4c36-96aa-8fab2b889709 None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:35.243655 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b09a1919-a09d-4c36-96aa-8fab2b889709 None None] PATH_INFO: `/` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:35.245564 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 156/319] 10.222.0.50 () {54 vars in 927 bytes} [Tue Sep 13 15:53:35 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:53:35.250751 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9dd8ed9-3209-4436-99fa-c7c88d33d08b None None] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:35.251083 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9dd8ed9-3209-4436-99fa-c7c88d33d08b None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:35.251365 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9dd8ed9-3209-4436-99fa-c7c88d33d08b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:35.258548 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:53:35.270721 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.auth.core [None req-b9dd8ed9-3209-4436-99fa-c7c88d33d08b None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81437) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:53:35.272242 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-b9dd8ed9-3209-4436-99fa-c7c88d33d08b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:53:35.341495 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 164/320] 10.222.0.50 () {58 vars in 1024 bytes} [Tue Sep 13 15:53:35 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 13 15:53:35.499256 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-383fcc60-0995-4c55-b67c-88bb8c55bcdd None None] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:35.499557 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-383fcc60-0995-4c55-b67c-88bb8c55bcdd None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:35.499811 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-383fcc60-0995-4c55-b67c-88bb8c55bcdd None None] PATH_INFO: `/` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:35.501286 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 157/321] 10.222.0.50 () {54 vars in 927 bytes} [Tue Sep 13 15:53:35 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:53:35.506200 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e74c95c-25a1-48b7-b6a7-6de6ba7d6a3a None None] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:35.506491 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e74c95c-25a1-48b7-b6a7-6de6ba7d6a3a None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:35.506766 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e74c95c-25a1-48b7-b6a7-6de6ba7d6a3a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:35.524822 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.auth.core [None req-9e74c95c-25a1-48b7-b6a7-6de6ba7d6a3a None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81437) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:53:35.526558 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-9e74c95c-25a1-48b7-b6a7-6de6ba7d6a3a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:53:35.599190 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 165/322] 10.222.0.50 () {58 vars in 1024 bytes} [Tue Sep 13 15:53:35 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 13 15:53:35.615670 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-383fcc60-0995-4c55-b67c-88bb8c55bcdd None None] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:53:35.684048 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:53:35.733402 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-383fcc60-0995-4c55-b67c-88bb8c55bcdd None None] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:53:35.734385 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61f9218a-2c6b-422b-a42f-a66ad1100857 None None] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:35.734665 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61f9218a-2c6b-422b-a42f-a66ad1100857 None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:35.734963 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61f9218a-2c6b-422b-a42f-a66ad1100857 None None] PATH_INFO: `/` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:35.736423 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 166/323] 10.222.0.50 () {54 vars in 927 bytes} [Tue Sep 13 15:53:35 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 13 15:53:35.741640 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbddbf1e-cf70-4aa5-ad4d-a943a347e183 None None] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:35.742149 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbddbf1e-cf70-4aa5-ad4d-a943a347e183 None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:35.742648 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbddbf1e-cf70-4aa5-ad4d-a943a347e183 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:35.761784 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.auth.core [None req-cbddbf1e-cf70-4aa5-ad4d-a943a347e183 None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81437) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:53:35.763107 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-cbddbf1e-cf70-4aa5-ad4d-a943a347e183 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:53:35.769672 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36e8e6be-b60e-4697-b8f0-278abaad0897 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:53:35.770304 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36e8e6be-b60e-4697-b8f0-278abaad0897 None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:35.770584 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36e8e6be-b60e-4697-b8f0-278abaad0897 None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:35.772745 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36e8e6be-b60e-4697-b8f0-278abaad0897 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:35.773888 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36e8e6be-b60e-4697-b8f0-278abaad0897 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:53:35.775387 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36e8e6be-b60e-4697-b8f0-278abaad0897 None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:53:35.777531 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 158/324] 10.222.0.50 () {56 vars in 1108 bytes} [Tue Sep 13 15:53:35 2022] GET /identity/v3/domains/default => generated 199 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:53:35.782945 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36e8e6be-b60e-4697-b8f0-278abaad0897 None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:53:35.830172 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:53:35.843452 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 167/325] 10.222.0.50 () {58 vars in 1024 bytes} [Tue Sep 13 15:53:35 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 13 15:53:35.875268 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbddbf1e-cf70-4aa5-ad4d-a943a347e183 None None] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:53:35.890712 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36e8e6be-b60e-4697-b8f0-278abaad0897 None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:53:35.915204 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:53:35.926545 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd63b68e-2684-4334-a775-bdc9eeac3c70 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:53:35.927179 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd63b68e-2684-4334-a775-bdc9eeac3c70 None admin] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:35.927455 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd63b68e-2684-4334-a775-bdc9eeac3c70 None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:35.927724 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd63b68e-2684-4334-a775-bdc9eeac3c70 None admin] PATH_INFO: `/v3/groups` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:35.928345 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd63b68e-2684-4334-a775-bdc9eeac3c70 None admin] RBAC: Authorizing `identity:create_group()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:53:35.929759 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd63b68e-2684-4334-a775-bdc9eeac3c70 None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:53:35.946887 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 159/326] 10.222.0.50 () {60 vars in 1134 bytes} [Tue Sep 13 15:53:35 2022] POST /identity/v3/groups => generated 226 bytes in 164 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 13 15:53:36.000357 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbddbf1e-cf70-4aa5-ad4d-a943a347e183 None None] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:53:36.042609 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d662732-d7a8-4d1e-a3f1-a4e51947a6b0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:53:36.042609 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d662732-d7a8-4d1e-a3f1-a4e51947a6b0 None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:36.042609 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d662732-d7a8-4d1e-a3f1-a4e51947a6b0 None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:36.042609 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d662732-d7a8-4d1e-a3f1-a4e51947a6b0 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:36.056674 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d662732-d7a8-4d1e-a3f1-a4e51947a6b0 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 13 15:53:36.063256 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application [None req-0d662732-d7a8-4d1e-a3f1-a4e51947a6b0 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 13 15:53:36.063256 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:53:36.063256 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:53:36.063256 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:53:36.063256 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:53:36.063256 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:53:36.063256 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:53:36.063256 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:53:36.063256 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:53:36.063256 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:53:36.063256 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:53:36.063256 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:53:36.063256 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 13 15:53:36.063256 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 13 15:53:36.063256 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 13 15:53:36.063256 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 13 15:53:36.064165 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 13 15:53:36.064165 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 13 15:53:36.064165 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 13 15:53:36.064165 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 13 15:53:36.064165 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:53:36.064165 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:53:36.064165 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:53:36.064165 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:53:36.064165 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:53:36.064165 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:53:36.064165 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:53:36.064165 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application with Lock( Sep 13 15:53:36.064165 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:53:36.064165 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:53:36.064165 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:53:36.064165 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:53:36.067746 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:53:36.067746 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:53:36.067746 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:53:36.067746 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:53:36.067746 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 13 15:53:36.067746 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 13 15:53:36.067746 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 13 15:53:36.067746 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 13 15:53:36.067746 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 13 15:53:36.067746 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 13 15:53:36.067746 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 13 15:53:36.067746 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application Sep 13 15:53:36.067746 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 168/327] 10.222.0.50 () {56 vars in 1099 bytes} [Tue Sep 13 15:53:35 2022] GET /identity/v3/roles/member => generated 84 bytes in 190 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 13 15:53:36.070836 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd63b68e-2684-4334-a775-bdc9eeac3c70 None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:53:36.112249 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d662732-d7a8-4d1e-a3f1-a4e51947a6b0 None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:53:36.180225 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:53:36.196518 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd63b68e-2684-4334-a775-bdc9eeac3c70 None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:53:36.234593 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d662732-d7a8-4d1e-a3f1-a4e51947a6b0 None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:53:36.247584 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fe56ee8-5e1f-458b-90b0-e1a4d047520c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:53:36.248204 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fe56ee8-5e1f-458b-90b0-e1a4d047520c None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:36.248480 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fe56ee8-5e1f-458b-90b0-e1a4d047520c None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:36.248747 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fe56ee8-5e1f-458b-90b0-e1a4d047520c None admin] PATH_INFO: `/v3/roles` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:36.249407 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fe56ee8-5e1f-458b-90b0-e1a4d047520c None admin] RBAC: Adding query filter params (name=member) {{(pid=81438) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:53:36.249832 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fe56ee8-5e1f-458b-90b0-e1a4d047520c None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:53:36.251228 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fe56ee8-5e1f-458b-90b0-e1a4d047520c None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:53:36.264468 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 160/328] 10.222.0.50 () {56 vars in 1101 bytes} [Tue Sep 13 15:53:36 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 194 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:53:36.269931 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fe56ee8-5e1f-458b-90b0-e1a4d047520c None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:53:36.283602 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95c62c6d-e2fc-47ac-a037-fb65da004636 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:53:36.284242 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95c62c6d-e2fc-47ac-a037-fb65da004636 None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:36.284518 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95c62c6d-e2fc-47ac-a037-fb65da004636 None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:36.285072 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95c62c6d-e2fc-47ac-a037-fb65da004636 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:36.297479 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95c62c6d-e2fc-47ac-a037-fb65da004636 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 13 15:53:36.302289 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application [None req-95c62c6d-e2fc-47ac-a037-fb65da004636 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Sep 13 15:53:36.302289 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:53:36.302289 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:53:36.302289 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:53:36.302289 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:53:36.302289 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:53:36.302289 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:53:36.302289 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:53:36.302289 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:53:36.302289 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:53:36.302289 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:53:36.302289 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:53:36.302289 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 13 15:53:36.302289 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 13 15:53:36.302289 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 13 15:53:36.302289 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 13 15:53:36.303231 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 13 15:53:36.303231 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 13 15:53:36.303231 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 13 15:53:36.303231 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 13 15:53:36.303231 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:53:36.303231 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:53:36.303231 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:53:36.303231 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:53:36.303231 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:53:36.303231 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:53:36.303231 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:53:36.303231 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application with Lock( Sep 13 15:53:36.303231 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:53:36.303231 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:53:36.303231 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:53:36.303231 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:53:36.304040 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:53:36.304040 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:53:36.304040 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:53:36.304040 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:53:36.304040 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 13 15:53:36.304040 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 13 15:53:36.304040 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 13 15:53:36.304040 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 13 15:53:36.304040 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 13 15:53:36.304040 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 13 15:53:36.304040 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Sep 13 15:53:36.304040 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application Sep 13 15:53:36.304040 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 169/329] 10.222.0.50 () {56 vars in 1099 bytes} [Tue Sep 13 15:53:36 2022] GET /identity/v3/roles/reader => generated 84 bytes in 192 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 13 15:53:36.312221 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95c62c6d-e2fc-47ac-a037-fb65da004636 None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:53:36.329594 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:53:36.410464 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fe56ee8-5e1f-458b-90b0-e1a4d047520c None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:53:36.444748 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95c62c6d-e2fc-47ac-a037-fb65da004636 None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:53:36.464420 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe3adbfc-5944-4ac8-bfb8-6d8a53f2d84b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:53:36.464650 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe3adbfc-5944-4ac8-bfb8-6d8a53f2d84b None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:36.465473 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:53:36.465598 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe3adbfc-5944-4ac8-bfb8-6d8a53f2d84b None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:36.465966 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe3adbfc-5944-4ac8-bfb8-6d8a53f2d84b None admin] PATH_INFO: `/v3/users/0878d73c1462483abf59bb21a38302a7` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:36.467492 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe3adbfc-5944-4ac8-bfb8-6d8a53f2d84b None admin] RBAC: Authorizing `identity:get_user(user_id=0878d73c1462483abf59bb21a38302a7)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:53:36.469883 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe3adbfc-5944-4ac8-bfb8-6d8a53f2d84b None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:53:36.471931 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 161/330] 10.222.0.50 () {56 vars in 1177 bytes} [Tue Sep 13 15:53:36 2022] GET /identity/v3/users/0878d73c1462483abf59bb21a38302a7 => generated 293 bytes in 203 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 13 15:53:36.478645 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe3adbfc-5944-4ac8-bfb8-6d8a53f2d84b None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:53:36.494020 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be9e395a-6d40-44f6-a47d-59fa79c74dd7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:53:36.495198 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be9e395a-6d40-44f6-a47d-59fa79c74dd7 None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:36.495645 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be9e395a-6d40-44f6-a47d-59fa79c74dd7 None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:36.496266 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be9e395a-6d40-44f6-a47d-59fa79c74dd7 None admin] PATH_INFO: `/v3/roles` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:36.496729 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be9e395a-6d40-44f6-a47d-59fa79c74dd7 None admin] RBAC: Adding query filter params (name=reader) {{(pid=81437) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:53:36.497031 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be9e395a-6d40-44f6-a47d-59fa79c74dd7 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:53:36.498447 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be9e395a-6d40-44f6-a47d-59fa79c74dd7 None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:53:36.509098 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 170/331] 10.222.0.50 () {56 vars in 1101 bytes} [Tue Sep 13 15:53:36 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 197 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:53:36.515177 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be9e395a-6d40-44f6-a47d-59fa79c74dd7 None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:53:36.611216 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe3adbfc-5944-4ac8-bfb8-6d8a53f2d84b None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:53:36.628967 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:53:36.654663 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af9c0181-70f7-4ad2-9b27-d0b49a0024c2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:53:36.655320 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af9c0181-70f7-4ad2-9b27-d0b49a0024c2 None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:36.655603 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af9c0181-70f7-4ad2-9b27-d0b49a0024c2 None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:36.656119 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af9c0181-70f7-4ad2-9b27-d0b49a0024c2 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:36.656846 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af9c0181-70f7-4ad2-9b27-d0b49a0024c2 None admin] RBAC: Adding query filter params (role.id=15db4aaca3264ae29bde38903ba43471, user.id=0878d73c1462483abf59bb21a38302a7, scope.system=all) {{(pid=81438) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:53:36.657162 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af9c0181-70f7-4ad2-9b27-d0b49a0024c2 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:53:36.657915 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be9e395a-6d40-44f6-a47d-59fa79c74dd7 None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:53:36.658629 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af9c0181-70f7-4ad2-9b27-d0b49a0024c2 None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:53:36.674047 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 162/332] 10.222.0.50 () {56 vars in 1308 bytes} [Tue Sep 13 15:53:36 2022] GET /identity/v3/role_assignments?user.id=0878d73c1462483abf59bb21a38302a7&scope.system=all&role.id=15db4aaca3264ae29bde38903ba43471 => generated 510 bytes in 196 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 13 15:53:36.707203 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4215400-283b-4bce-b975-1ac3cbcd0b9c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:53:36.707787 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4215400-283b-4bce-b975-1ac3cbcd0b9c None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:36.708068 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4215400-283b-4bce-b975-1ac3cbcd0b9c None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:36.708339 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4215400-283b-4bce-b975-1ac3cbcd0b9c None admin] PATH_INFO: `/v3/users/8f5df6f7cab84a5bb067824eb0b7c534` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:36.709776 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4215400-283b-4bce-b975-1ac3cbcd0b9c None admin] RBAC: Authorizing `identity:get_user(user_id=8f5df6f7cab84a5bb067824eb0b7c534)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:53:36.711341 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4215400-283b-4bce-b975-1ac3cbcd0b9c None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:53:36.713014 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 171/333] 10.222.0.50 () {56 vars in 1177 bytes} [Tue Sep 13 15:53:36 2022] GET /identity/v3/users/8f5df6f7cab84a5bb067824eb0b7c534 => generated 293 bytes in 198 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 13 15:53:36.719577 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af9c0181-70f7-4ad2-9b27-d0b49a0024c2 None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:53:36.858529 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af9c0181-70f7-4ad2-9b27-d0b49a0024c2 None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:53:36.902663 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7269a95d-0ff6-4ff6-b109-61c18714f239 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:53:36.903384 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7269a95d-0ff6-4ff6-b109-61c18714f239 None admin] REQUEST_METHOD: `PUT` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:36.903664 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7269a95d-0ff6-4ff6-b109-61c18714f239 None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:36.903941 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7269a95d-0ff6-4ff6-b109-61c18714f239 None admin] PATH_INFO: `/v3/system/users/8f5df6f7cab84a5bb067824eb0b7c534/roles/b2d1670446a54085ad2bdb935f003d25` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:36.905392 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7269a95d-0ff6-4ff6-b109-61c18714f239 None admin] RBAC: Authorizing `identity:create_system_grant_for_user(user_id=8f5df6f7cab84a5bb067824eb0b7c534, role_id=b2d1670446a54085ad2bdb935f003d25)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:53:36.906775 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7269a95d-0ff6-4ff6-b109-61c18714f239 None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:53:36.919515 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 163/334] 10.222.0.50 () {58 vars in 1334 bytes} [Tue Sep 13 15:53:36 2022] PUT /identity/v3/system/users/8f5df6f7cab84a5bb067824eb0b7c534/roles/b2d1670446a54085ad2bdb935f003d25 => generated 0 bytes in 200 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:53:38.628982 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaccaba4-74ea-48ad-a3d5-6f06257e859c None None] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:38.629522 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaccaba4-74ea-48ad-a3d5-6f06257e859c None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:38.629522 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaccaba4-74ea-48ad-a3d5-6f06257e859c None None] PATH_INFO: `/` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:38.631040 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 172/335] 10.222.0.50 () {54 vars in 927 bytes} [Tue Sep 13 15:53:38 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:53:38.636701 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd2c34b1-29c3-402c-ae32-56c0fe81c8a9 None None] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:38.637141 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd2c34b1-29c3-402c-ae32-56c0fe81c8a9 None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:38.637314 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd2c34b1-29c3-402c-ae32-56c0fe81c8a9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:38.668642 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.auth.core [None req-fd2c34b1-29c3-402c-ae32-56c0fe81c8a9 None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81438) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:53:38.669957 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-fd2c34b1-29c3-402c-ae32-56c0fe81c8a9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:53:38.713123 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:53:38.763407 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 164/336] 10.222.0.50 () {58 vars in 1024 bytes} [Tue Sep 13 15:53:38 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 128 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 13 15:53:38.792846 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1fd6a47-7ee1-4bd0-9248-5d957ac90591 None None] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:38.793212 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1fd6a47-7ee1-4bd0-9248-5d957ac90591 None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:38.793402 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1fd6a47-7ee1-4bd0-9248-5d957ac90591 None None] PATH_INFO: `/` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:38.795073 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 173/337] 10.222.0.50 () {54 vars in 927 bytes} [Tue Sep 13 15:53:38 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:53:38.806884 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c0a30c9-0fdf-478f-b3c3-8d8a76d1631e None None] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:38.807248 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c0a30c9-0fdf-478f-b3c3-8d8a76d1631e None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:38.807792 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c0a30c9-0fdf-478f-b3c3-8d8a76d1631e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:38.832884 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.auth.core [None req-6c0a30c9-0fdf-478f-b3c3-8d8a76d1631e None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81438) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:53:38.832884 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-6c0a30c9-0fdf-478f-b3c3-8d8a76d1631e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:53:38.918588 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 165/338] 10.222.0.50 () {58 vars in 1024 bytes} [Tue Sep 13 15:53:38 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 112 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 13 15:53:39.356427 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1fd6a47-7ee1-4bd0-9248-5d957ac90591 None None] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:53:39.413166 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:53:39.474218 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1fd6a47-7ee1-4bd0-9248-5d957ac90591 None None] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:53:39.511248 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23d965af-b72d-4fb0-b333-afa04fd2e147 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:53:39.511891 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23d965af-b72d-4fb0-b333-afa04fd2e147 None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:39.512169 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23d965af-b72d-4fb0-b333-afa04fd2e147 None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:39.512443 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23d965af-b72d-4fb0-b333-afa04fd2e147 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:39.518140 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c0a30c9-0fdf-478f-b3c3-8d8a76d1631e None None] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:53:39.531323 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23d965af-b72d-4fb0-b333-afa04fd2e147 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 13 15:53:39.536200 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application [None req-23d965af-b72d-4fb0-b333-afa04fd2e147 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 13 15:53:39.536200 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:53:39.536200 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:53:39.536200 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:53:39.536200 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:53:39.536200 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:53:39.536200 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:53:39.536200 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:53:39.536200 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:53:39.536200 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:53:39.536200 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:53:39.536200 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:53:39.536200 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 13 15:53:39.536200 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 13 15:53:39.536200 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 13 15:53:39.536200 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 13 15:53:39.537065 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 13 15:53:39.537065 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 13 15:53:39.537065 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 13 15:53:39.537065 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 13 15:53:39.537065 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:53:39.537065 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:53:39.537065 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:53:39.537065 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:53:39.537065 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:53:39.537065 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:53:39.537065 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:53:39.537065 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application with Lock( Sep 13 15:53:39.537065 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:53:39.537065 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:53:39.537065 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:53:39.537065 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:53:39.537840 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:53:39.537840 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:53:39.537840 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:53:39.537840 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:53:39.537840 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 13 15:53:39.537840 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 13 15:53:39.537840 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 13 15:53:39.537840 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 13 15:53:39.537840 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 13 15:53:39.537840 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 13 15:53:39.537840 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 13 15:53:39.537840 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application Sep 13 15:53:39.537840 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 174/339] 10.222.0.50 () {56 vars in 1099 bytes} [Tue Sep 13 15:53:39 2022] GET /identity/v3/roles/member => generated 84 bytes in 181 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:53:39.544572 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23d965af-b72d-4fb0-b333-afa04fd2e147 None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:53:39.735354 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:53:39.739981 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c0a30c9-0fdf-478f-b3c3-8d8a76d1631e None None] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:53:39.774169 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23d965af-b72d-4fb0-b333-afa04fd2e147 None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:53:39.781821 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:53:39.820235 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bc788a7-b33f-4f03-8965-5b630793645e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:53:39.820893 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bc788a7-b33f-4f03-8965-5b630793645e None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:39.821176 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bc788a7-b33f-4f03-8965-5b630793645e None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:39.821449 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bc788a7-b33f-4f03-8965-5b630793645e None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:39.841746 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5bc788a7-b33f-4f03-8965-5b630793645e None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 13 15:53:39.846879 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application [None req-5bc788a7-b33f-4f03-8965-5b630793645e None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 13 15:53:39.846879 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:53:39.846879 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:53:39.846879 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:53:39.846879 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:53:39.846879 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:53:39.846879 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:53:39.846879 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:53:39.846879 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:53:39.846879 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:53:39.846879 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:53:39.846879 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:53:39.846879 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 13 15:53:39.846879 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 13 15:53:39.846879 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 13 15:53:39.846879 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 13 15:53:39.847836 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 13 15:53:39.847836 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 13 15:53:39.847836 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 13 15:53:39.847836 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 13 15:53:39.847836 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:53:39.847836 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:53:39.847836 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:53:39.847836 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:53:39.847836 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:53:39.847836 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:53:39.847836 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:53:39.847836 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application with Lock( Sep 13 15:53:39.847836 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:53:39.847836 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:53:39.847836 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:53:39.847836 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:53:39.848632 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:53:39.848632 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:53:39.848632 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:53:39.848632 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:53:39.848632 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 13 15:53:39.848632 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 13 15:53:39.848632 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 13 15:53:39.848632 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 13 15:53:39.848632 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 13 15:53:39.848632 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 13 15:53:39.848632 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 13 15:53:39.848632 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Sep 13 15:53:39.850015 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 166/340] 10.222.0.50 () {56 vars in 1114 bytes} [Tue Sep 13 15:53:39 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 331 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:53:39.852020 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0dfcc71c-0f78-4697-92ff-53a7e12ae7a2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:53:39.852984 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dfcc71c-0f78-4697-92ff-53a7e12ae7a2 None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:39.853274 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dfcc71c-0f78-4697-92ff-53a7e12ae7a2 None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:39.853548 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dfcc71c-0f78-4697-92ff-53a7e12ae7a2 None admin] PATH_INFO: `/v3/roles` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:39.854251 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0dfcc71c-0f78-4697-92ff-53a7e12ae7a2 None admin] RBAC: Adding query filter params (name=member) {{(pid=81437) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:53:39.854555 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0dfcc71c-0f78-4697-92ff-53a7e12ae7a2 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:53:39.855967 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0dfcc71c-0f78-4697-92ff-53a7e12ae7a2 None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:53:39.858413 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bc788a7-b33f-4f03-8965-5b630793645e None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:53:39.875073 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 175/341] 10.222.0.50 () {56 vars in 1101 bytes} [Tue Sep 13 15:53:39 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 327 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:53:39.876555 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0dfcc71c-0f78-4697-92ff-53a7e12ae7a2 None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:53:40.071019 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bc788a7-b33f-4f03-8965-5b630793645e None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:53:40.088880 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0dfcc71c-0f78-4697-92ff-53a7e12ae7a2 None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:53:40.127572 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-989dad48-7f6e-4051-869a-1705e1b36afb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:53:40.128783 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-989dad48-7f6e-4051-869a-1705e1b36afb None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:40.129220 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-989dad48-7f6e-4051-869a-1705e1b36afb None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:40.129503 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-989dad48-7f6e-4051-869a-1705e1b36afb None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:40.152303 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7018783c-92d5-4ceb-9b9a-653763aae244 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:53:40.153014 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7018783c-92d5-4ceb-9b9a-653763aae244 None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:40.153300 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7018783c-92d5-4ceb-9b9a-653763aae244 None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:40.153577 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7018783c-92d5-4ceb-9b9a-653763aae244 None admin] PATH_INFO: `/v3/roles` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:40.154257 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7018783c-92d5-4ceb-9b9a-653763aae244 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81438) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:53:40.154565 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7018783c-92d5-4ceb-9b9a-653763aae244 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:53:40.156022 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7018783c-92d5-4ceb-9b9a-653763aae244 None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:53:40.179290 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:53:40.185235 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 167/342] 10.222.0.50 () {56 vars in 1111 bytes} [Tue Sep 13 15:53:39 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 327 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:53:40.186555 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b741e6d9-ec28-4605-9b5b-e6fba09b43f3 None None] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:40.186844 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b741e6d9-ec28-4605-9b5b-e6fba09b43f3 None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:40.187148 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b741e6d9-ec28-4605-9b5b-e6fba09b43f3 None None] PATH_INFO: `/` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:40.188609 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 168/343] 10.222.0.50 () {54 vars in 927 bytes} [Tue Sep 13 15:53:40 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:53:40.194184 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b741e6d9-ec28-4605-9b5b-e6fba09b43f3 None None] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:53:40.216828 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-989dad48-7f6e-4051-869a-1705e1b36afb None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:53:40.218628 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-989dad48-7f6e-4051-869a-1705e1b36afb None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:53:40.236387 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:53:40.353399 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 176/344] 10.222.0.50 () {58 vars in 1282 bytes} [Tue Sep 13 15:53:39 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 477 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 13 15:53:40.355693 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89a42b94-0f98-473a-b5d4-f2699805e147 None None] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:40.356285 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89a42b94-0f98-473a-b5d4-f2699805e147 None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:40.357347 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89a42b94-0f98-473a-b5d4-f2699805e147 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:40.381831 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.auth.core [None req-89a42b94-0f98-473a-b5d4-f2699805e147 None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81437) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:53:40.383932 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-89a42b94-0f98-473a-b5d4-f2699805e147 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:53:40.395627 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b741e6d9-ec28-4605-9b5b-e6fba09b43f3 None None] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:53:40.444059 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa1bc394-6519-42fd-b31e-826cbc457e1a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:53:40.444059 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa1bc394-6519-42fd-b31e-826cbc457e1a None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:40.444350 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa1bc394-6519-42fd-b31e-826cbc457e1a None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:40.444507 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa1bc394-6519-42fd-b31e-826cbc457e1a None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:40.478380 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:53:40.498433 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 177/345] 10.222.0.50 () {58 vars in 1024 bytes} [Tue Sep 13 15:53:40 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 144 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 13 15:53:40.500538 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecad60a3-82fc-49b4-96b2-19e1554e878a None None] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:40.502053 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecad60a3-82fc-49b4-96b2-19e1554e878a None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:40.502702 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecad60a3-82fc-49b4-96b2-19e1554e878a None None] PATH_INFO: `/` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:40.504363 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 178/346] 10.222.0.50 () {54 vars in 927 bytes} [Tue Sep 13 15:53:40 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:53:40.505411 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecad60a3-82fc-49b4-96b2-19e1554e878a None None] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:53:40.556110 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa1bc394-6519-42fd-b31e-826cbc457e1a None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:53:40.558725 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa1bc394-6519-42fd-b31e-826cbc457e1a None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:53:40.579987 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:53:40.630576 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecad60a3-82fc-49b4-96b2-19e1554e878a None None] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:53:40.669938 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25322942-7768-4e8d-b791-750f05df4f46 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:53:40.671003 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25322942-7768-4e8d-b791-750f05df4f46 None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:40.671467 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25322942-7768-4e8d-b791-750f05df4f46 None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:40.671994 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25322942-7768-4e8d-b791-750f05df4f46 None admin] PATH_INFO: `/v3/projects/423edf83b2b24540924e990bb324af1e` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:40.673396 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25322942-7768-4e8d-b791-750f05df4f46 None admin] RBAC: Authorizing `identity:get_project(project_id=423edf83b2b24540924e990bb324af1e)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:53:40.675176 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25322942-7768-4e8d-b791-750f05df4f46 None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:53:40.677529 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 179/347] 10.222.0.50 () {56 vars in 1186 bytes} [Tue Sep 13 15:53:40 2022] GET /identity/v3/projects/423edf83b2b24540924e990bb324af1e => generated 298 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:53:40.678945 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1754d8b7-4195-4322-8b31-9afd41e3a054 None None] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:40.679420 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1754d8b7-4195-4322-8b31-9afd41e3a054 None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:40.679964 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1754d8b7-4195-4322-8b31-9afd41e3a054 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:40.697274 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 169/348] 10.222.0.50 () {58 vars in 1282 bytes} [Tue Sep 13 15:53:40 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 503 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Sep 13 15:53:40.699264 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1ea8947-616c-460b-b321-248bf1ba87bb None None] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:40.699552 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1ea8947-616c-460b-b321-248bf1ba87bb None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:40.699825 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1ea8947-616c-460b-b321-248bf1ba87bb None None] PATH_INFO: `/` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:40.701076 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 170/349] 10.222.0.50 () {54 vars in 927 bytes} [Tue Sep 13 15:53:40 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:53:40.701822 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1ea8947-616c-460b-b321-248bf1ba87bb None None] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:53:40.704227 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.auth.core [None req-1754d8b7-4195-4322-8b31-9afd41e3a054 None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81437) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:53:40.705687 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-1754d8b7-4195-4322-8b31-9afd41e3a054 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:53:40.785417 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:53:40.798196 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 180/350] 10.222.0.50 () {58 vars in 1024 bytes} [Tue Sep 13 15:53:40 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 120 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 13 15:53:40.799159 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1754d8b7-4195-4322-8b31-9afd41e3a054 None None] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:53:40.837829 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1ea8947-616c-460b-b321-248bf1ba87bb None None] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:53:40.859231 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:53:40.881179 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38f843ff-0134-420f-941e-963b96d8915d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:53:40.882292 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38f843ff-0134-420f-941e-963b96d8915d None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:40.882770 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38f843ff-0134-420f-941e-963b96d8915d None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:40.883415 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38f843ff-0134-420f-941e-963b96d8915d None admin] PATH_INFO: `/v3/groups/acdd1f9701134c2e9b5d957895b30027` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:40.892382 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38f843ff-0134-420f-941e-963b96d8915d None admin] RBAC: Authorizing `identity:get_group(group_id=acdd1f9701134c2e9b5d957895b30027)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:53:40.894411 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38f843ff-0134-420f-941e-963b96d8915d None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:53:40.896513 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 171/351] 10.222.0.50 () {56 vars in 1180 bytes} [Tue Sep 13 15:53:40 2022] GET /identity/v3/groups/acdd1f9701134c2e9b5d957895b30027 => generated 226 bytes in 195 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:53:40.898172 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-543b5e2b-2aeb-4821-9fae-0f3e61c900ba None None] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:40.898657 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-543b5e2b-2aeb-4821-9fae-0f3e61c900ba None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:40.899154 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-543b5e2b-2aeb-4821-9fae-0f3e61c900ba None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:40.920315 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.auth.core [None req-543b5e2b-2aeb-4821-9fae-0f3e61c900ba None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81438) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:53:40.921475 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1754d8b7-4195-4322-8b31-9afd41e3a054 None None] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:53:40.922141 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-543b5e2b-2aeb-4821-9fae-0f3e61c900ba None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:53:40.969082 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-531ea267-9618-490e-8ab3-b50ca2dbbb10 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:53:40.969728 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-531ea267-9618-490e-8ab3-b50ca2dbbb10 None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:40.970027 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-531ea267-9618-490e-8ab3-b50ca2dbbb10 None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:40.970291 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-531ea267-9618-490e-8ab3-b50ca2dbbb10 None admin] PATH_INFO: `/v3/projects/423edf83b2b24540924e990bb324af1e` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:40.971441 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-531ea267-9618-490e-8ab3-b50ca2dbbb10 None admin] RBAC: Authorizing `identity:get_project(project_id=423edf83b2b24540924e990bb324af1e)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:53:40.973335 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-531ea267-9618-490e-8ab3-b50ca2dbbb10 None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:53:40.977151 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 181/352] 10.222.0.50 () {56 vars in 1186 bytes} [Tue Sep 13 15:53:40 2022] GET /identity/v3/projects/423edf83b2b24540924e990bb324af1e => generated 298 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:53:40.977941 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-531ea267-9618-490e-8ab3-b50ca2dbbb10 None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:53:41.006252 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 172/353] 10.222.0.50 () {58 vars in 1024 bytes} [Tue Sep 13 15:53:40 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 13 15:53:41.007552 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28e2a286-4898-4b44-acc7-ccbc62d3bbe5 None None] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:41.007841 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28e2a286-4898-4b44-acc7-ccbc62d3bbe5 None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:41.008119 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28e2a286-4898-4b44-acc7-ccbc62d3bbe5 None None] PATH_INFO: `/` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:41.009357 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 173/354] 10.222.0.50 () {54 vars in 927 bytes} [Tue Sep 13 15:53:41 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:53:41.010051 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28e2a286-4898-4b44-acc7-ccbc62d3bbe5 None None] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:53:41.066385 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:53:41.109901 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-531ea267-9618-490e-8ab3-b50ca2dbbb10 None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:53:41.141168 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28e2a286-4898-4b44-acc7-ccbc62d3bbe5 None None] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:53:41.150657 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e95e82b2-dab7-455d-baad-0512a3af8ec6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:53:41.151934 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e95e82b2-dab7-455d-baad-0512a3af8ec6 None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:41.152418 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e95e82b2-dab7-455d-baad-0512a3af8ec6 None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:41.152889 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e95e82b2-dab7-455d-baad-0512a3af8ec6 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:41.153919 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e95e82b2-dab7-455d-baad-0512a3af8ec6 None admin] RBAC: Adding query filter params (group.id=acdd1f9701134c2e9b5d957895b30027, role.id=15db4aaca3264ae29bde38903ba43471, scope.project.id=423edf83b2b24540924e990bb324af1e) {{(pid=81437) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:53:41.154485 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e95e82b2-dab7-455d-baad-0512a3af8ec6 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:53:41.156214 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:53:41.156731 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e95e82b2-dab7-455d-baad-0512a3af8ec6 None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:53:41.168199 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 182/355] 10.222.0.50 () {56 vars in 1376 bytes} [Tue Sep 13 15:53:40 2022] GET /identity/v3/role_assignments?group.id=acdd1f9701134c2e9b5d957895b30027&scope.project.id=423edf83b2b24540924e990bb324af1e&role.id=15db4aaca3264ae29bde38903ba43471 => generated 260 bytes in 190 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:53:41.170173 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e95e82b2-dab7-455d-baad-0512a3af8ec6 None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:53:41.189297 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a7d4fb7-c259-4994-a794-9190c153e67e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:53:41.190474 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a7d4fb7-c259-4994-a794-9190c153e67e None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:41.190758 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a7d4fb7-c259-4994-a794-9190c153e67e None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:41.191060 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a7d4fb7-c259-4994-a794-9190c153e67e None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:41.207811 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a7d4fb7-c259-4994-a794-9190c153e67e None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 13 15:53:41.214654 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application [None req-9a7d4fb7-c259-4994-a794-9190c153e67e None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Sep 13 15:53:41.214654 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:53:41.214654 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:53:41.214654 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:53:41.214654 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:53:41.214654 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:53:41.214654 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:53:41.214654 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:53:41.214654 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:53:41.214654 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:53:41.214654 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:53:41.214654 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:53:41.214654 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 13 15:53:41.214654 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 13 15:53:41.214654 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 13 15:53:41.214654 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 13 15:53:41.215707 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 13 15:53:41.215707 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 13 15:53:41.215707 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 13 15:53:41.215707 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 13 15:53:41.215707 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:53:41.215707 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:53:41.215707 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:53:41.215707 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:53:41.215707 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:53:41.215707 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:53:41.215707 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:53:41.215707 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application with Lock( Sep 13 15:53:41.215707 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:53:41.215707 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:53:41.215707 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:53:41.215707 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:53:41.217143 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:53:41.217143 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:53:41.217143 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:53:41.217143 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:53:41.217143 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 13 15:53:41.217143 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 13 15:53:41.217143 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 13 15:53:41.217143 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 13 15:53:41.217143 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 13 15:53:41.217143 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 13 15:53:41.217143 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Sep 13 15:53:41.217143 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Sep 13 15:53:41.217143 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 174/356] 10.222.0.50 () {56 vars in 1099 bytes} [Tue Sep 13 15:53:41 2022] GET /identity/v3/roles/reader => generated 84 bytes in 206 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:53:41.217819 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06137cc4-f334-4c99-9836-a31bf0f6b5f8 None None] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:41.218065 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06137cc4-f334-4c99-9836-a31bf0f6b5f8 None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:41.218295 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06137cc4-f334-4c99-9836-a31bf0f6b5f8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:41.239227 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.auth.core [None req-06137cc4-f334-4c99-9836-a31bf0f6b5f8 None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81438) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:53:41.240450 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-06137cc4-f334-4c99-9836-a31bf0f6b5f8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:53:41.311191 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e95e82b2-dab7-455d-baad-0512a3af8ec6 None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:53:41.328835 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 175/357] 10.222.0.50 () {58 vars in 1024 bytes} [Tue Sep 13 15:53:41 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 112 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 13 15:53:41.329614 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06137cc4-f334-4c99-9836-a31bf0f6b5f8 None None] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:53:41.344710 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:53:41.356410 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-501fb4ec-ac24-4a21-a830-45818dc70b84 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:53:41.357420 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-501fb4ec-ac24-4a21-a830-45818dc70b84 None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:41.357908 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-501fb4ec-ac24-4a21-a830-45818dc70b84 None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:41.358425 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-501fb4ec-ac24-4a21-a830-45818dc70b84 None admin] PATH_INFO: `/v3/groups/acdd1f9701134c2e9b5d957895b30027` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:41.361326 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-501fb4ec-ac24-4a21-a830-45818dc70b84 None admin] RBAC: Authorizing `identity:get_group(group_id=acdd1f9701134c2e9b5d957895b30027)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:53:41.363144 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-501fb4ec-ac24-4a21-a830-45818dc70b84 None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:53:41.365043 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 183/358] 10.222.0.50 () {56 vars in 1180 bytes} [Tue Sep 13 15:53:41 2022] GET /identity/v3/groups/acdd1f9701134c2e9b5d957895b30027 => generated 226 bytes in 196 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:53:41.366121 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-501fb4ec-ac24-4a21-a830-45818dc70b84 None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:53:41.479940 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06137cc4-f334-4c99-9836-a31bf0f6b5f8 None None] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:53:41.487196 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:53:41.501241 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-501fb4ec-ac24-4a21-a830-45818dc70b84 None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:53:41.518518 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4cc9607-5552-4e79-9355-cb0e52abeaac None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:53:41.519217 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4cc9607-5552-4e79-9355-cb0e52abeaac None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:41.519495 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4cc9607-5552-4e79-9355-cb0e52abeaac None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:41.519767 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4cc9607-5552-4e79-9355-cb0e52abeaac None admin] PATH_INFO: `/v3/roles/member` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:41.531179 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4cc9607-5552-4e79-9355-cb0e52abeaac None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 13 15:53:41.536294 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application [None req-b4cc9607-5552-4e79-9355-cb0e52abeaac None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 13 15:53:41.536294 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:53:41.536294 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:53:41.536294 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:53:41.536294 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:53:41.536294 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:53:41.536294 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:53:41.536294 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:53:41.536294 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:53:41.536294 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:53:41.536294 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:53:41.536294 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:53:41.536294 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 13 15:53:41.536294 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 13 15:53:41.536294 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 13 15:53:41.536294 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 13 15:53:41.537184 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 13 15:53:41.537184 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 13 15:53:41.537184 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 13 15:53:41.537184 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 13 15:53:41.537184 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:53:41.537184 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:53:41.537184 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:53:41.537184 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:53:41.537184 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:53:41.537184 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:53:41.537184 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:53:41.537184 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application with Lock( Sep 13 15:53:41.537184 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:53:41.537184 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:53:41.537184 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:53:41.537184 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:53:41.537973 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:53:41.537973 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:53:41.537973 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:53:41.537973 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:53:41.537973 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 13 15:53:41.537973 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 13 15:53:41.537973 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 13 15:53:41.537973 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 13 15:53:41.537973 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 13 15:53:41.537973 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 13 15:53:41.537973 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 13 15:53:41.537973 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Sep 13 15:53:41.537973 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 176/359] 10.222.0.50 () {56 vars in 1099 bytes} [Tue Sep 13 15:53:41 2022] GET /identity/v3/roles/member => generated 84 bytes in 208 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:53:41.538698 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4cc9607-5552-4e79-9355-cb0e52abeaac None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:53:41.540561 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24504539-17f2-447b-9daa-d0e295e45576 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:53:41.541175 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24504539-17f2-447b-9daa-d0e295e45576 None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:41.541483 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24504539-17f2-447b-9daa-d0e295e45576 None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:41.541755 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24504539-17f2-447b-9daa-d0e295e45576 None admin] PATH_INFO: `/v3/roles` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:41.542417 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24504539-17f2-447b-9daa-d0e295e45576 None admin] RBAC: Adding query filter params (name=reader) {{(pid=81437) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:53:41.542718 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24504539-17f2-447b-9daa-d0e295e45576 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:53:41.544571 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24504539-17f2-447b-9daa-d0e295e45576 None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:53:41.554573 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 184/360] 10.222.0.50 () {56 vars in 1101 bytes} [Tue Sep 13 15:53:41 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:53:41.555590 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24504539-17f2-447b-9daa-d0e295e45576 None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:53:41.621684 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:53:41.656129 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4cc9607-5552-4e79-9355-cb0e52abeaac None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:53:41.670861 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24504539-17f2-447b-9daa-d0e295e45576 None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:53:41.699039 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0c9cfa7-5d36-4151-8ee8-896513006fac None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:53:41.699701 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0c9cfa7-5d36-4151-8ee8-896513006fac None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:41.699980 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0c9cfa7-5d36-4151-8ee8-896513006fac None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:41.700248 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0c9cfa7-5d36-4151-8ee8-896513006fac None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:41.711630 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-888b78aa-d84a-4af9-bf41-4bae44503d69 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:53:41.712248 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-888b78aa-d84a-4af9-bf41-4bae44503d69 None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:41.712588 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-888b78aa-d84a-4af9-bf41-4bae44503d69 None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:41.712883 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-888b78aa-d84a-4af9-bf41-4bae44503d69 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:41.713782 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-888b78aa-d84a-4af9-bf41-4bae44503d69 None admin] RBAC: Adding query filter params (group.id=acdd1f9701134c2e9b5d957895b30027, role.id=f74e3f6c20bb426ca030f211c3a9aef5, scope.project.id=423edf83b2b24540924e990bb324af1e) {{(pid=81437) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:53:41.714096 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-888b78aa-d84a-4af9-bf41-4bae44503d69 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:53:41.715896 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-888b78aa-d84a-4af9-bf41-4bae44503d69 None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:53:41.717429 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0c9cfa7-5d36-4151-8ee8-896513006fac None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 13 15:53:41.722725 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application [None req-b0c9cfa7-5d36-4151-8ee8-896513006fac None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 13 15:53:41.722725 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:53:41.722725 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:53:41.722725 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:53:41.722725 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:53:41.722725 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:53:41.722725 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:53:41.722725 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:53:41.722725 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:53:41.722725 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:53:41.722725 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:53:41.722725 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:53:41.722725 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 13 15:53:41.722725 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 13 15:53:41.722725 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 13 15:53:41.722725 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 13 15:53:41.723756 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 13 15:53:41.723756 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 13 15:53:41.723756 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 13 15:53:41.723756 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 13 15:53:41.723756 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:53:41.723756 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:53:41.723756 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:53:41.723756 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:53:41.723756 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:53:41.723756 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:53:41.723756 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:53:41.723756 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application with Lock( Sep 13 15:53:41.723756 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:53:41.723756 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:53:41.723756 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:53:41.723756 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:53:41.724606 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:53:41.724606 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:53:41.724606 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:53:41.724606 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:53:41.724606 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 13 15:53:41.724606 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 13 15:53:41.724606 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 13 15:53:41.724606 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 13 15:53:41.724606 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 13 15:53:41.724606 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 13 15:53:41.724606 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 13 15:53:41.724606 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Sep 13 15:53:41.724606 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 177/361] 10.222.0.50 () {56 vars in 1114 bytes} [Tue Sep 13 15:53:41 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 186 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:53:41.725348 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0c9cfa7-5d36-4151-8ee8-896513006fac None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:53:41.733646 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 185/362] 10.222.0.50 () {56 vars in 1376 bytes} [Tue Sep 13 15:53:41 2022] GET /identity/v3/role_assignments?group.id=acdd1f9701134c2e9b5d957895b30027&scope.project.id=423edf83b2b24540924e990bb324af1e&role.id=f74e3f6c20bb426ca030f211c3a9aef5 => generated 260 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:53:41.734411 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-888b78aa-d84a-4af9-bf41-4bae44503d69 None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:53:41.767143 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:53:41.890841 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0c9cfa7-5d36-4151-8ee8-896513006fac None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:53:41.911445 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-888b78aa-d84a-4af9-bf41-4bae44503d69 None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:53:41.945784 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:53:41.951733 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-beb1dd47-862e-4fb2-8541-3731a6b98c37 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:53:41.953320 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-beb1dd47-862e-4fb2-8541-3731a6b98c37 None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:41.953320 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-beb1dd47-862e-4fb2-8541-3731a6b98c37 None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:41.953320 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-beb1dd47-862e-4fb2-8541-3731a6b98c37 None admin] PATH_INFO: `/v3/roles` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:41.954587 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-beb1dd47-862e-4fb2-8541-3731a6b98c37 None admin] RBAC: Adding query filter params (name=member) {{(pid=81438) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:53:41.954587 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-beb1dd47-862e-4fb2-8541-3731a6b98c37 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:53:41.955901 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-beb1dd47-862e-4fb2-8541-3731a6b98c37 None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:53:41.966848 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 178/363] 10.222.0.50 () {56 vars in 1101 bytes} [Tue Sep 13 15:53:41 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 242 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:53:41.967756 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-beb1dd47-862e-4fb2-8541-3731a6b98c37 None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:53:41.971967 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1fe9f97-12fb-44a3-8a76-00fc12e6f7de None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:53:41.973745 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1fe9f97-12fb-44a3-8a76-00fc12e6f7de None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:41.974234 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1fe9f97-12fb-44a3-8a76-00fc12e6f7de None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:41.974759 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1fe9f97-12fb-44a3-8a76-00fc12e6f7de None admin] PATH_INFO: `/v3/users/8f5df6f7cab84a5bb067824eb0b7c534` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:41.976234 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1fe9f97-12fb-44a3-8a76-00fc12e6f7de None admin] RBAC: Authorizing `identity:get_user(user_id=8f5df6f7cab84a5bb067824eb0b7c534)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:53:41.977913 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1fe9f97-12fb-44a3-8a76-00fc12e6f7de None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:53:41.980948 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 186/364] 10.222.0.50 () {56 vars in 1177 bytes} [Tue Sep 13 15:53:41 2022] GET /identity/v3/users/8f5df6f7cab84a5bb067824eb0b7c534 => generated 293 bytes in 247 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:53:41.982365 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1fe9f97-12fb-44a3-8a76-00fc12e6f7de None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:53:42.096346 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:53:42.116193 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-beb1dd47-862e-4fb2-8541-3731a6b98c37 None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:53:42.125804 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1fe9f97-12fb-44a3-8a76-00fc12e6f7de None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:53:42.154060 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b23e017-0740-418c-bbe5-32abb7fe9ccd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:53:42.155151 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b23e017-0740-418c-bbe5-32abb7fe9ccd None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:42.155666 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b23e017-0740-418c-bbe5-32abb7fe9ccd None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:42.156163 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b23e017-0740-418c-bbe5-32abb7fe9ccd None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:42.164816 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-461a6e21-086c-4962-9583-c491a74f542d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:53:42.165396 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-461a6e21-086c-4962-9583-c491a74f542d None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:42.165728 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-461a6e21-086c-4962-9583-c491a74f542d None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:42.166007 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-461a6e21-086c-4962-9583-c491a74f542d None admin] PATH_INFO: `/v3/roles` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:42.168177 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-461a6e21-086c-4962-9583-c491a74f542d None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81437) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:53:42.168177 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-461a6e21-086c-4962-9583-c491a74f542d None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:53:42.168931 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-461a6e21-086c-4962-9583-c491a74f542d None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:53:42.171145 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b23e017-0740-418c-bbe5-32abb7fe9ccd None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 13 15:53:42.176907 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application [None req-8b23e017-0740-418c-bbe5-32abb7fe9ccd None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 13 15:53:42.176907 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:53:42.176907 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:53:42.176907 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:53:42.176907 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:53:42.176907 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:53:42.176907 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:53:42.176907 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:53:42.176907 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:53:42.176907 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:53:42.176907 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:53:42.176907 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:53:42.176907 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 13 15:53:42.176907 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 13 15:53:42.176907 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 13 15:53:42.176907 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 13 15:53:42.177828 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 13 15:53:42.177828 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 13 15:53:42.177828 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 13 15:53:42.177828 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 13 15:53:42.177828 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:53:42.177828 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:53:42.177828 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:53:42.177828 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:53:42.177828 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:53:42.177828 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:53:42.177828 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:53:42.177828 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application with Lock( Sep 13 15:53:42.177828 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:53:42.177828 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:53:42.177828 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:53:42.177828 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:53:42.178629 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:53:42.178629 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:53:42.178629 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:53:42.178629 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:53:42.178629 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 13 15:53:42.178629 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 13 15:53:42.178629 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 13 15:53:42.178629 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 13 15:53:42.178629 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 13 15:53:42.178629 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 13 15:53:42.178629 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 13 15:53:42.178629 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Sep 13 15:53:42.180944 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 179/365] 10.222.0.50 () {56 vars in 1096 bytes} [Tue Sep 13 15:53:41 2022] GET /identity/v3/roles/admin => generated 83 bytes in 214 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:53:42.181978 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b23e017-0740-418c-bbe5-32abb7fe9ccd None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:53:42.183846 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 187/366] 10.222.0.50 () {56 vars in 1111 bytes} [Tue Sep 13 15:53:41 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 202 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:53:42.184807 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-461a6e21-086c-4962-9583-c491a74f542d None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:53:42.240903 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:53:42.300216 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-461a6e21-086c-4962-9583-c491a74f542d None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:53:42.305069 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b23e017-0740-418c-bbe5-32abb7fe9ccd None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:53:42.339340 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09ef34b3-7018-4889-ac5d-b319b876cf02 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:53:42.340447 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09ef34b3-7018-4889-ac5d-b319b876cf02 None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:42.340941 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09ef34b3-7018-4889-ac5d-b319b876cf02 None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:42.341504 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09ef34b3-7018-4889-ac5d-b319b876cf02 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:42.342437 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09ef34b3-7018-4889-ac5d-b319b876cf02 None admin] RBAC: Adding query filter params (role.id=b2d1670446a54085ad2bdb935f003d25, user.id=8f5df6f7cab84a5bb067824eb0b7c534, scope.system=all) {{(pid=81437) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:53:42.343066 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09ef34b3-7018-4889-ac5d-b319b876cf02 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:53:42.344452 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13965ca7-6bc8-49e2-944a-8484a9cc3b27 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:53:42.345043 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09ef34b3-7018-4889-ac5d-b319b876cf02 None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:53:42.345158 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13965ca7-6bc8-49e2-944a-8484a9cc3b27 None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:42.345342 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13965ca7-6bc8-49e2-944a-8484a9cc3b27 None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:42.345595 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13965ca7-6bc8-49e2-944a-8484a9cc3b27 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:42.359895 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 188/367] 10.222.0.50 () {56 vars in 1308 bytes} [Tue Sep 13 15:53:42 2022] GET /identity/v3/role_assignments?user.id=8f5df6f7cab84a5bb067824eb0b7c534&scope.system=all&role.id=b2d1670446a54085ad2bdb935f003d25 => generated 510 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:53:42.361045 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09ef34b3-7018-4889-ac5d-b319b876cf02 None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:53:42.367507 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:53:42.428929 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13965ca7-6bc8-49e2-944a-8484a9cc3b27 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:53:42.430545 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13965ca7-6bc8-49e2-944a-8484a9cc3b27 None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:53:42.550313 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 180/368] 10.222.0.50 () {58 vars in 1282 bytes} [Tue Sep 13 15:53:42 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 368 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 13 15:53:42.551289 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13965ca7-6bc8-49e2-944a-8484a9cc3b27 None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:53:42.552846 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09ef34b3-7018-4889-ac5d-b319b876cf02 None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:53:42.571348 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:53:42.609632 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:53:42.621066 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efd470f0-e1b5-436a-b1f4-78e945b81bd1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:53:42.621859 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efd470f0-e1b5-436a-b1f4-78e945b81bd1 None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:42.622242 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efd470f0-e1b5-436a-b1f4-78e945b81bd1 None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:42.622831 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efd470f0-e1b5-436a-b1f4-78e945b81bd1 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:42.676437 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13965ca7-6bc8-49e2-944a-8484a9cc3b27 None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:53:42.741108 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63b376a9-bddd-4cdb-8b3f-f87a35fea358 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:53:42.741737 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63b376a9-bddd-4cdb-8b3f-f87a35fea358 None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:42.742017 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63b376a9-bddd-4cdb-8b3f-f87a35fea358 None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:42.742298 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63b376a9-bddd-4cdb-8b3f-f87a35fea358 None admin] PATH_INFO: `/v3/roles` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:42.743343 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63b376a9-bddd-4cdb-8b3f-f87a35fea358 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81438) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:53:42.743644 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63b376a9-bddd-4cdb-8b3f-f87a35fea358 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:53:42.745261 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63b376a9-bddd-4cdb-8b3f-f87a35fea358 None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:53:42.750582 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efd470f0-e1b5-436a-b1f4-78e945b81bd1 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:53:42.751185 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efd470f0-e1b5-436a-b1f4-78e945b81bd1 None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:53:42.759760 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 181/369] 10.222.0.50 () {56 vars in 1099 bytes} [Tue Sep 13 15:53:42 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 209 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:53:42.761000 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63b376a9-bddd-4cdb-8b3f-f87a35fea358 None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:53:42.892424 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63b376a9-bddd-4cdb-8b3f-f87a35fea358 None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:53:42.896313 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 189/370] 10.222.0.50 () {58 vars in 1282 bytes} [Tue Sep 13 15:53:42 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 535 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 13 15:53:42.896746 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efd470f0-e1b5-436a-b1f4-78e945b81bd1 None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:53:42.901392 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:53:42.910959 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:53:42.935821 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6d4e921-453f-4440-bacd-3cf428b2387e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:53:42.936977 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6d4e921-453f-4440-bacd-3cf428b2387e None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:42.937477 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6d4e921-453f-4440-bacd-3cf428b2387e None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:42.937948 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6d4e921-453f-4440-bacd-3cf428b2387e None admin] PATH_INFO: `/v3/projects/9ec1adca55204df2b653531664b60123` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:42.939372 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6d4e921-453f-4440-bacd-3cf428b2387e None admin] RBAC: Authorizing `identity:get_project(project_id=9ec1adca55204df2b653531664b60123)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:53:42.941034 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6d4e921-453f-4440-bacd-3cf428b2387e None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:53:42.942887 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 182/371] 10.222.0.50 () {56 vars in 1186 bytes} [Tue Sep 13 15:53:42 2022] GET /identity/v3/projects/9ec1adca55204df2b653531664b60123 => generated 302 bytes in 183 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:53:42.943984 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6d4e921-453f-4440-bacd-3cf428b2387e None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:53:43.019446 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efd470f0-e1b5-436a-b1f4-78e945b81bd1 None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:53:43.065555 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6d4e921-453f-4440-bacd-3cf428b2387e None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:53:43.069333 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2ceddc7-e0c9-4dff-91d8-5231160c5eec None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:53:43.069961 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2ceddc7-e0c9-4dff-91d8-5231160c5eec None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:43.070239 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2ceddc7-e0c9-4dff-91d8-5231160c5eec None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:43.070512 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2ceddc7-e0c9-4dff-91d8-5231160c5eec None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:43.104568 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30e1de19-0919-4873-b054-f1d13190ed26 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:53:43.105579 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30e1de19-0919-4873-b054-f1d13190ed26 None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:43.106073 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30e1de19-0919-4873-b054-f1d13190ed26 None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:43.106558 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30e1de19-0919-4873-b054-f1d13190ed26 None admin] PATH_INFO: `/v3/projects/9ec1adca55204df2b653531664b60123` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:43.108043 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30e1de19-0919-4873-b054-f1d13190ed26 None admin] RBAC: Authorizing `identity:get_project(project_id=9ec1adca55204df2b653531664b60123)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:53:43.109689 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30e1de19-0919-4873-b054-f1d13190ed26 None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:53:43.111927 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 183/372] 10.222.0.50 () {56 vars in 1186 bytes} [Tue Sep 13 15:53:42 2022] GET /identity/v3/projects/9ec1adca55204df2b653531664b60123 => generated 302 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:53:43.112940 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30e1de19-0919-4873-b054-f1d13190ed26 None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:53:43.177361 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2ceddc7-e0c9-4dff-91d8-5231160c5eec None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:53:43.178733 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2ceddc7-e0c9-4dff-91d8-5231160c5eec None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:53:43.181288 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:53:43.190668 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:53:43.253790 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30e1de19-0919-4873-b054-f1d13190ed26 None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:53:43.296124 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e2f9150-4e1e-4022-9e3e-fe044029e8e9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:53:43.296581 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 190/373] 10.222.0.50 () {58 vars in 1282 bytes} [Tue Sep 13 15:53:42 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 400 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 13 15:53:43.297059 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e2f9150-4e1e-4022-9e3e-fe044029e8e9 None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:43.297341 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e2f9150-4e1e-4022-9e3e-fe044029e8e9 None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:43.297511 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2ceddc7-e0c9-4dff-91d8-5231160c5eec None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:53:43.297612 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e2f9150-4e1e-4022-9e3e-fe044029e8e9 None admin] PATH_INFO: `/v3/groups/acdd1f9701134c2e9b5d957895b30027` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:43.303415 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e2f9150-4e1e-4022-9e3e-fe044029e8e9 None admin] RBAC: Authorizing `identity:get_group(group_id=acdd1f9701134c2e9b5d957895b30027)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:53:43.305821 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e2f9150-4e1e-4022-9e3e-fe044029e8e9 None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:53:43.308304 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 184/374] 10.222.0.50 () {56 vars in 1180 bytes} [Tue Sep 13 15:53:43 2022] GET /identity/v3/groups/acdd1f9701134c2e9b5d957895b30027 => generated 226 bytes in 196 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:53:43.309091 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e2f9150-4e1e-4022-9e3e-fe044029e8e9 None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:53:43.417305 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2ceddc7-e0c9-4dff-91d8-5231160c5eec None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:53:43.427240 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:53:43.429304 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e2f9150-4e1e-4022-9e3e-fe044029e8e9 None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:53:43.441611 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:53:43.460094 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0360588-2765-4299-b0d0-7b6cc493faaf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:53:43.460720 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0360588-2765-4299-b0d0-7b6cc493faaf None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:43.461097 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0360588-2765-4299-b0d0-7b6cc493faaf None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:43.461430 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0360588-2765-4299-b0d0-7b6cc493faaf None admin] PATH_INFO: `/v3/groups/acdd1f9701134c2e9b5d957895b30027` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:43.462985 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0360588-2765-4299-b0d0-7b6cc493faaf None admin] RBAC: Authorizing `identity:get_group(group_id=acdd1f9701134c2e9b5d957895b30027)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:53:43.465202 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0360588-2765-4299-b0d0-7b6cc493faaf None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:53:43.466871 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 191/375] 10.222.0.50 () {56 vars in 1180 bytes} [Tue Sep 13 15:53:43 2022] GET /identity/v3/groups/acdd1f9701134c2e9b5d957895b30027 => generated 226 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:53:43.468661 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0360588-2765-4299-b0d0-7b6cc493faaf None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:53:43.475354 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-076af324-0482-4dea-87c5-821012e3f763 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:53:43.476004 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-076af324-0482-4dea-87c5-821012e3f763 None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:43.476287 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-076af324-0482-4dea-87c5-821012e3f763 None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:43.476559 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-076af324-0482-4dea-87c5-821012e3f763 None admin] PATH_INFO: `/v3/projects/555bee08a3fa4d08bdd8ecc881adc45b` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:43.477709 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-076af324-0482-4dea-87c5-821012e3f763 None admin] RBAC: Authorizing `identity:get_project(project_id=555bee08a3fa4d08bdd8ecc881adc45b)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:53:43.479414 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-076af324-0482-4dea-87c5-821012e3f763 None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:53:43.481062 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 185/376] 10.222.0.50 () {56 vars in 1186 bytes} [Tue Sep 13 15:53:43 2022] GET /identity/v3/projects/555bee08a3fa4d08bdd8ecc881adc45b => generated 344 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:53:43.481834 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-076af324-0482-4dea-87c5-821012e3f763 None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:53:43.587063 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0360588-2765-4299-b0d0-7b6cc493faaf None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:53:43.605514 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-076af324-0482-4dea-87c5-821012e3f763 None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:53:43.632363 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0976d607-1320-4801-a5cb-7d419323dd02 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:53:43.632980 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0976d607-1320-4801-a5cb-7d419323dd02 None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:43.633254 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0976d607-1320-4801-a5cb-7d419323dd02 None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:43.633522 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0976d607-1320-4801-a5cb-7d419323dd02 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:43.634247 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0976d607-1320-4801-a5cb-7d419323dd02 None admin] RBAC: Adding query filter params (group.id=acdd1f9701134c2e9b5d957895b30027, role.id=15db4aaca3264ae29bde38903ba43471, scope.project.id=9ec1adca55204df2b653531664b60123) {{(pid=81437) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:53:43.634650 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0976d607-1320-4801-a5cb-7d419323dd02 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:53:43.636840 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0976d607-1320-4801-a5cb-7d419323dd02 None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:53:43.647070 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 192/377] 10.222.0.50 () {56 vars in 1376 bytes} [Tue Sep 13 15:53:43 2022] GET /identity/v3/role_assignments?group.id=acdd1f9701134c2e9b5d957895b30027&scope.project.id=9ec1adca55204df2b653531664b60123&role.id=15db4aaca3264ae29bde38903ba43471 => generated 260 bytes in 180 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:53:43.647899 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c144741d-36a3-41bd-a12a-8509f12bdd88 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:53:43.648035 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0976d607-1320-4801-a5cb-7d419323dd02 None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:53:43.648527 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c144741d-36a3-41bd-a12a-8509f12bdd88 None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:43.648812 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c144741d-36a3-41bd-a12a-8509f12bdd88 None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:43.649090 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c144741d-36a3-41bd-a12a-8509f12bdd88 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:43.649837 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c144741d-36a3-41bd-a12a-8509f12bdd88 None admin] RBAC: Adding query filter params (group.id=acdd1f9701134c2e9b5d957895b30027, role.id=f74e3f6c20bb426ca030f211c3a9aef5, scope.project.id=9ec1adca55204df2b653531664b60123) {{(pid=81438) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:53:43.650151 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c144741d-36a3-41bd-a12a-8509f12bdd88 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:53:43.652303 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c144741d-36a3-41bd-a12a-8509f12bdd88 None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:53:43.658678 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 186/378] 10.222.0.50 () {56 vars in 1376 bytes} [Tue Sep 13 15:53:43 2022] GET /identity/v3/role_assignments?group.id=acdd1f9701134c2e9b5d957895b30027&scope.project.id=9ec1adca55204df2b653531664b60123&role.id=f74e3f6c20bb426ca030f211c3a9aef5 => generated 260 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:53:43.724703 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:53:43.731949 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34b86ab2-1700-4fa7-943d-39ab330dedb4 None None] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:43.732292 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34b86ab2-1700-4fa7-943d-39ab330dedb4 None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:43.732508 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34b86ab2-1700-4fa7-943d-39ab330dedb4 None None] PATH_INFO: `/` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:43.733927 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 187/379] 10.222.0.50 () {54 vars in 927 bytes} [Tue Sep 13 15:53:43 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 13 15:53:43.739678 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c3f40c4-34c0-40f8-8d0d-c112d4363440 None None] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:43.740001 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c3f40c4-34c0-40f8-8d0d-c112d4363440 None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:43.740240 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c3f40c4-34c0-40f8-8d0d-c112d4363440 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:43.763697 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.auth.core [None req-4c3f40c4-34c0-40f8-8d0d-c112d4363440 None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81438) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:53:43.764948 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-4c3f40c4-34c0-40f8-8d0d-c112d4363440 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:53:43.817058 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:53:43.822271 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0976d607-1320-4801-a5cb-7d419323dd02 None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:53:43.882597 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 188/380] 10.222.0.50 () {58 vars in 1024 bytes} [Tue Sep 13 15:53:43 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 144 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 13 15:53:43.891429 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdf48414-4aab-49d7-b2f8-d7ff32a46d15 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:53:43.892089 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdf48414-4aab-49d7-b2f8-d7ff32a46d15 None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:43.892366 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdf48414-4aab-49d7-b2f8-d7ff32a46d15 None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:43.892638 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdf48414-4aab-49d7-b2f8-d7ff32a46d15 None admin] PATH_INFO: `/v3/groups/58a500e5c4724a8eb4916f4251241715` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:43.912085 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdf48414-4aab-49d7-b2f8-d7ff32a46d15 None admin] RBAC: Authorizing `identity:get_group(group_id=58a500e5c4724a8eb4916f4251241715)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:53:43.913594 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdf48414-4aab-49d7-b2f8-d7ff32a46d15 None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:53:43.915898 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 193/381] 10.222.0.50 () {56 vars in 1180 bytes} [Tue Sep 13 15:53:43 2022] GET /identity/v3/groups/58a500e5c4724a8eb4916f4251241715 => generated 229 bytes in 268 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:53:43.923339 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c3f40c4-34c0-40f8-8d0d-c112d4363440 None None] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:53:44.048796 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c3f40c4-34c0-40f8-8d0d-c112d4363440 None None] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:53:44.093981 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ea0c766-ec18-42be-8d2f-db3a1afede75 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:53:44.094597 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ea0c766-ec18-42be-8d2f-db3a1afede75 None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:44.094873 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ea0c766-ec18-42be-8d2f-db3a1afede75 None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:44.095165 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ea0c766-ec18-42be-8d2f-db3a1afede75 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:44.095898 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ea0c766-ec18-42be-8d2f-db3a1afede75 None admin] RBAC: Adding query filter params (group.id=58a500e5c4724a8eb4916f4251241715, role.id=031d4e1a23e34f71858fc4a6dfbd4ccb, scope.project.id=555bee08a3fa4d08bdd8ecc881adc45b) {{(pid=81438) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:53:44.096209 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ea0c766-ec18-42be-8d2f-db3a1afede75 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:53:44.097635 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ea0c766-ec18-42be-8d2f-db3a1afede75 None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:53:44.106893 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 189/382] 10.222.0.50 () {56 vars in 1376 bytes} [Tue Sep 13 15:53:43 2022] GET /identity/v3/role_assignments?group.id=58a500e5c4724a8eb4916f4251241715&scope.project.id=555bee08a3fa4d08bdd8ecc881adc45b&role.id=031d4e1a23e34f71858fc4a6dfbd4ccb => generated 260 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:53:44.346347 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdf48414-4aab-49d7-b2f8-d7ff32a46d15 None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:53:44.435085 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dad609d-14d5-4468-a16a-3d162b46e974 None None] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:44.435395 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dad609d-14d5-4468-a16a-3d162b46e974 None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:44.435650 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dad609d-14d5-4468-a16a-3d162b46e974 None None] PATH_INFO: `/` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:44.436970 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 190/383] 10.222.0.50 () {54 vars in 927 bytes} [Tue Sep 13 15:53:44 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 13 15:53:44.443166 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bd341b7-d3ed-4e6a-bf98-9da5ed3ebe3b None None] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:44.443444 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bd341b7-d3ed-4e6a-bf98-9da5ed3ebe3b None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:44.443721 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bd341b7-d3ed-4e6a-bf98-9da5ed3ebe3b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:44.467325 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.auth.core [None req-9bd341b7-d3ed-4e6a-bf98-9da5ed3ebe3b None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81438) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:53:44.468513 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-9bd341b7-d3ed-4e6a-bf98-9da5ed3ebe3b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:53:44.479832 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdf48414-4aab-49d7-b2f8-d7ff32a46d15 None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:53:44.493040 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:53:44.531097 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:53:44.550318 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 191/384] 10.222.0.50 () {58 vars in 1024 bytes} [Tue Sep 13 15:53:44 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 13 15:53:44.554440 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2da5a05-db0d-4316-82e3-cb63c68398d8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:53:44.555120 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2da5a05-db0d-4316-82e3-cb63c68398d8 None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:44.555404 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2da5a05-db0d-4316-82e3-cb63c68398d8 None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:44.555677 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2da5a05-db0d-4316-82e3-cb63c68398d8 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:44.579933 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2da5a05-db0d-4316-82e3-cb63c68398d8 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 13 15:53:44.585116 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application [None req-a2da5a05-db0d-4316-82e3-cb63c68398d8 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 13 15:53:44.585116 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:53:44.585116 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:53:44.585116 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:53:44.585116 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:53:44.585116 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:53:44.585116 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:53:44.585116 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:53:44.585116 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:53:44.585116 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:53:44.585116 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:53:44.585116 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:53:44.585116 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 13 15:53:44.585116 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 13 15:53:44.585116 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 13 15:53:44.585116 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 13 15:53:44.585985 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 13 15:53:44.585985 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 13 15:53:44.585985 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 13 15:53:44.585985 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 13 15:53:44.585985 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:53:44.585985 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:53:44.585985 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:53:44.585985 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:53:44.585985 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:53:44.585985 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:53:44.585985 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:53:44.585985 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application with Lock( Sep 13 15:53:44.585985 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:53:44.585985 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:53:44.585985 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:53:44.585985 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:53:44.586780 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:53:44.586780 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:53:44.586780 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:53:44.586780 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:53:44.586780 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 13 15:53:44.586780 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 13 15:53:44.586780 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 13 15:53:44.586780 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 13 15:53:44.586780 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 13 15:53:44.586780 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 13 15:53:44.586780 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 13 15:53:44.586780 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application Sep 13 15:53:44.589673 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 194/385] 10.222.0.50 () {56 vars in 1099 bytes} [Tue Sep 13 15:53:44 2022] GET /identity/v3/roles/member => generated 84 bytes in 244 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:53:44.595589 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bd341b7-d3ed-4e6a-bf98-9da5ed3ebe3b None None] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:53:44.747970 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bd341b7-d3ed-4e6a-bf98-9da5ed3ebe3b None None] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:53:44.793489 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2242a41-ab2e-4d4e-a68c-ec3fb571768d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:53:44.794345 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2242a41-ab2e-4d4e-a68c-ec3fb571768d None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:44.794854 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2242a41-ab2e-4d4e-a68c-ec3fb571768d None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:44.795298 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2242a41-ab2e-4d4e-a68c-ec3fb571768d None admin] PATH_INFO: `/v3/roles` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:44.796107 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2242a41-ab2e-4d4e-a68c-ec3fb571768d None admin] RBAC: Adding query filter params (name=member) {{(pid=81438) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:53:44.796580 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2242a41-ab2e-4d4e-a68c-ec3fb571768d None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:53:44.798096 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2242a41-ab2e-4d4e-a68c-ec3fb571768d None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:53:44.807139 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 192/386] 10.222.0.50 () {56 vars in 1101 bytes} [Tue Sep 13 15:53:44 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 212 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:53:44.813192 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2da5a05-db0d-4316-82e3-cb63c68398d8 None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:53:44.908401 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2242a41-ab2e-4d4e-a68c-ec3fb571768d None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:53:44.961652 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2da5a05-db0d-4316-82e3-cb63c68398d8 None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:53:44.971254 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:53:45.031154 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34900f30-5540-406e-a16a-6d0f8ace994b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:53:45.031890 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34900f30-5540-406e-a16a-6d0f8ace994b None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:45.032253 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34900f30-5540-406e-a16a-6d0f8ace994b None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:45.032628 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34900f30-5540-406e-a16a-6d0f8ace994b None admin] PATH_INFO: `/v3/groups/acdd1f9701134c2e9b5d957895b30027` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:45.034492 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34900f30-5540-406e-a16a-6d0f8ace994b None admin] RBAC: Authorizing `identity:get_group(group_id=acdd1f9701134c2e9b5d957895b30027)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:53:45.036174 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34900f30-5540-406e-a16a-6d0f8ace994b None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:53:45.046656 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 195/387] 10.222.0.50 () {56 vars in 1180 bytes} [Tue Sep 13 15:53:44 2022] GET /identity/v3/groups/acdd1f9701134c2e9b5d957895b30027 => generated 226 bytes in 225 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:53:45.048790 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34900f30-5540-406e-a16a-6d0f8ace994b None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:53:45.061137 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2242a41-ab2e-4d4e-a68c-ec3fb571768d None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:53:45.098511 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f82ee998-4ff6-4aef-9045-208114d813dd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:53:45.099531 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f82ee998-4ff6-4aef-9045-208114d813dd None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:45.100316 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f82ee998-4ff6-4aef-9045-208114d813dd None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:45.100762 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f82ee998-4ff6-4aef-9045-208114d813dd None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:45.114708 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f82ee998-4ff6-4aef-9045-208114d813dd None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 13 15:53:45.120934 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:53:45.122945 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application [None req-f82ee998-4ff6-4aef-9045-208114d813dd None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 13 15:53:45.122945 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:53:45.122945 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:53:45.122945 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:53:45.122945 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:53:45.122945 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:53:45.122945 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:53:45.122945 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:53:45.122945 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:53:45.122945 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:53:45.122945 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:53:45.122945 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:53:45.122945 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 13 15:53:45.122945 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 13 15:53:45.122945 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 13 15:53:45.122945 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 13 15:53:45.124159 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 13 15:53:45.124159 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 13 15:53:45.124159 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 13 15:53:45.124159 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 13 15:53:45.124159 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:53:45.124159 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:53:45.124159 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:53:45.124159 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:53:45.124159 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:53:45.124159 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:53:45.124159 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:53:45.124159 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application with Lock( Sep 13 15:53:45.124159 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:53:45.124159 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:53:45.124159 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:53:45.124159 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:53:45.125200 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:53:45.125200 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:53:45.125200 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:53:45.125200 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:53:45.125200 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 13 15:53:45.125200 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 13 15:53:45.125200 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 13 15:53:45.125200 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 13 15:53:45.125200 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 13 15:53:45.125200 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 13 15:53:45.125200 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 13 15:53:45.125200 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Sep 13 15:53:45.127487 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 193/388] 10.222.0.50 () {56 vars in 1114 bytes} [Tue Sep 13 15:53:44 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 220 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 13 15:53:45.133000 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f82ee998-4ff6-4aef-9045-208114d813dd None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:53:45.249769 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f82ee998-4ff6-4aef-9045-208114d813dd None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:53:45.256170 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34900f30-5540-406e-a16a-6d0f8ace994b None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:53:45.267350 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:53:45.293158 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26d586e0-5f0f-489b-885e-bb0718fbec44 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:53:45.294073 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26d586e0-5f0f-489b-885e-bb0718fbec44 None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:45.294548 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26d586e0-5f0f-489b-885e-bb0718fbec44 None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:45.295034 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26d586e0-5f0f-489b-885e-bb0718fbec44 None admin] PATH_INFO: `/v3/roles` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:45.296009 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26d586e0-5f0f-489b-885e-bb0718fbec44 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81438) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:53:45.296494 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26d586e0-5f0f-489b-885e-bb0718fbec44 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:53:45.299190 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26d586e0-5f0f-489b-885e-bb0718fbec44 None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:53:45.310008 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 194/389] 10.222.0.50 () {56 vars in 1111 bytes} [Tue Sep 13 15:53:45 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 13 15:53:45.315830 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26d586e0-5f0f-489b-885e-bb0718fbec44 None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:53:45.325383 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c6e3812-fcd3-4782-a002-837797650d25 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:53:45.326030 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c6e3812-fcd3-4782-a002-837797650d25 None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:45.326306 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c6e3812-fcd3-4782-a002-837797650d25 None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:45.326577 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c6e3812-fcd3-4782-a002-837797650d25 None admin] PATH_INFO: `/v3/projects/423edf83b2b24540924e990bb324af1e` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:45.331603 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c6e3812-fcd3-4782-a002-837797650d25 None admin] RBAC: Authorizing `identity:get_project(project_id=423edf83b2b24540924e990bb324af1e)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:53:45.333066 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c6e3812-fcd3-4782-a002-837797650d25 None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:53:45.335032 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 196/390] 10.222.0.50 () {56 vars in 1186 bytes} [Tue Sep 13 15:53:45 2022] GET /identity/v3/projects/423edf83b2b24540924e990bb324af1e => generated 298 bytes in 286 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:53:45.363860 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c6e3812-fcd3-4782-a002-837797650d25 None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:53:45.437077 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26d586e0-5f0f-489b-885e-bb0718fbec44 None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:53:45.475725 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9ebfef6-c511-4257-b1fa-30920ecafb6f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:53:45.476655 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9ebfef6-c511-4257-b1fa-30920ecafb6f None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:45.477103 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9ebfef6-c511-4257-b1fa-30920ecafb6f None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:45.477622 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9ebfef6-c511-4257-b1fa-30920ecafb6f None admin] PATH_INFO: `/v3/groups/acdd1f9701134c2e9b5d957895b30027` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:45.479624 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9ebfef6-c511-4257-b1fa-30920ecafb6f None admin] RBAC: Authorizing `identity:get_group(group_id=acdd1f9701134c2e9b5d957895b30027)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:53:45.481743 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9ebfef6-c511-4257-b1fa-30920ecafb6f None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:53:45.484257 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 195/391] 10.222.0.50 () {56 vars in 1180 bytes} [Tue Sep 13 15:53:45 2022] GET /identity/v3/groups/acdd1f9701134c2e9b5d957895b30027 => generated 226 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 13 15:53:45.489440 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9ebfef6-c511-4257-b1fa-30920ecafb6f None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:53:45.507294 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:53:45.510958 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c6e3812-fcd3-4782-a002-837797650d25 None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:53:45.517747 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:53:45.552449 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5726f1af-f742-4d1b-ac3e-fa2b7bd3beec None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:53:45.553045 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5726f1af-f742-4d1b-ac3e-fa2b7bd3beec None admin] REQUEST_METHOD: `PUT` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:45.553322 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5726f1af-f742-4d1b-ac3e-fa2b7bd3beec None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:45.553641 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5726f1af-f742-4d1b-ac3e-fa2b7bd3beec None admin] PATH_INFO: `/v3/projects/423edf83b2b24540924e990bb324af1e/groups/acdd1f9701134c2e9b5d957895b30027/roles/15db4aaca3264ae29bde38903ba43471` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:45.555769 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5726f1af-f742-4d1b-ac3e-fa2b7bd3beec None admin] RBAC: Authorizing `identity:create_grant(project_id=423edf83b2b24540924e990bb324af1e, group_id=acdd1f9701134c2e9b5d957895b30027, role_id=15db4aaca3264ae29bde38903ba43471)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:53:45.557357 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5726f1af-f742-4d1b-ac3e-fa2b7bd3beec None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:53:45.574852 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 197/392] 10.222.0.50 () {58 vars in 1442 bytes} [Tue Sep 13 15:53:45 2022] PUT /identity/v3/projects/423edf83b2b24540924e990bb324af1e/groups/acdd1f9701134c2e9b5d957895b30027/roles/15db4aaca3264ae29bde38903ba43471 => generated 0 bytes in 211 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:53:45.624216 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9ebfef6-c511-4257-b1fa-30920ecafb6f None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:53:45.667259 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-636aed4e-3b24-487a-af39-90616de72ae6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:53:45.667908 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-636aed4e-3b24-487a-af39-90616de72ae6 None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:45.668185 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-636aed4e-3b24-487a-af39-90616de72ae6 None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:45.668456 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-636aed4e-3b24-487a-af39-90616de72ae6 None admin] PATH_INFO: `/v3/projects/423edf83b2b24540924e990bb324af1e` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:45.669636 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-636aed4e-3b24-487a-af39-90616de72ae6 None admin] RBAC: Authorizing `identity:get_project(project_id=423edf83b2b24540924e990bb324af1e)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:53:45.671122 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-636aed4e-3b24-487a-af39-90616de72ae6 None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:53:45.673201 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 196/393] 10.222.0.50 () {56 vars in 1186 bytes} [Tue Sep 13 15:53:45 2022] GET /identity/v3/projects/423edf83b2b24540924e990bb324af1e => generated 298 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:53:45.679022 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5726f1af-f742-4d1b-ac3e-fa2b7bd3beec None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:53:45.806585 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5726f1af-f742-4d1b-ac3e-fa2b7bd3beec None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:53:45.841545 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cde0d809-6c03-4684-9094-af36eb5216c4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:53:45.842209 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cde0d809-6c03-4684-9094-af36eb5216c4 None admin] REQUEST_METHOD: `PUT` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:45.842484 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cde0d809-6c03-4684-9094-af36eb5216c4 None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:45.842753 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cde0d809-6c03-4684-9094-af36eb5216c4 None admin] PATH_INFO: `/v3/projects/423edf83b2b24540924e990bb324af1e/groups/acdd1f9701134c2e9b5d957895b30027/roles/f74e3f6c20bb426ca030f211c3a9aef5` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:45.844879 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cde0d809-6c03-4684-9094-af36eb5216c4 None admin] RBAC: Authorizing `identity:create_grant(project_id=423edf83b2b24540924e990bb324af1e, group_id=acdd1f9701134c2e9b5d957895b30027, role_id=f74e3f6c20bb426ca030f211c3a9aef5)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:53:45.846315 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cde0d809-6c03-4684-9094-af36eb5216c4 None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:53:45.860839 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 198/394] 10.222.0.50 () {58 vars in 1442 bytes} [Tue Sep 13 15:53:45 2022] PUT /identity/v3/projects/423edf83b2b24540924e990bb324af1e/groups/acdd1f9701134c2e9b5d957895b30027/roles/f74e3f6c20bb426ca030f211c3a9aef5 => generated 0 bytes in 182 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:53:46.492185 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47f19d0b-d489-4d87-a9a2-b3b4c7c89059 None None] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:46.492694 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47f19d0b-d489-4d87-a9a2-b3b4c7c89059 None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:46.492792 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47f19d0b-d489-4d87-a9a2-b3b4c7c89059 None None] PATH_INFO: `/` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:46.494295 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 197/395] 10.222.0.50 () {54 vars in 927 bytes} [Tue Sep 13 15:53:46 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:53:46.500229 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a568ef4-9c69-4981-b585-d0f2339e8f1a None None] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:46.500641 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a568ef4-9c69-4981-b585-d0f2339e8f1a None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:46.500808 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a568ef4-9c69-4981-b585-d0f2339e8f1a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:46.520982 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.auth.core [None req-2a568ef4-9c69-4981-b585-d0f2339e8f1a None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81437) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:53:46.522170 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-2a568ef4-9c69-4981-b585-d0f2339e8f1a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:53:46.539954 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:53:46.617500 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 199/396] 10.222.0.50 () {58 vars in 1024 bytes} [Tue Sep 13 15:53:46 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 118 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 13 15:53:46.733080 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f696825-a7a1-4db2-af1d-93fafa5ec785 None None] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:46.733362 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f696825-a7a1-4db2-af1d-93fafa5ec785 None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:46.733646 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f696825-a7a1-4db2-af1d-93fafa5ec785 None None] PATH_INFO: `/` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:46.734986 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 198/397] 10.222.0.50 () {54 vars in 927 bytes} [Tue Sep 13 15:53:46 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 13 15:53:46.740635 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-438f3bdb-6913-4d57-98a7-0408db3e3129 None None] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:46.740915 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-438f3bdb-6913-4d57-98a7-0408db3e3129 None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:46.741191 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-438f3bdb-6913-4d57-98a7-0408db3e3129 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:46.768027 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.auth.core [None req-438f3bdb-6913-4d57-98a7-0408db3e3129 None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81437) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:53:46.769154 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-438f3bdb-6913-4d57-98a7-0408db3e3129 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:53:46.848842 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 200/398] 10.222.0.50 () {58 vars in 1024 bytes} [Tue Sep 13 15:53:46 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 13 15:53:46.942150 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f696825-a7a1-4db2-af1d-93fafa5ec785 None None] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:53:47.047240 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:53:47.069971 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f696825-a7a1-4db2-af1d-93fafa5ec785 None None] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:53:47.108033 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bd11c77-ead4-4165-967e-bdf33a3d44a9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:53:47.110651 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bd11c77-ead4-4165-967e-bdf33a3d44a9 None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:47.110651 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bd11c77-ead4-4165-967e-bdf33a3d44a9 None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:47.110651 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bd11c77-ead4-4165-967e-bdf33a3d44a9 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:47.130562 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4bd11c77-ead4-4165-967e-bdf33a3d44a9 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 13 15:53:47.139768 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application [None req-4bd11c77-ead4-4165-967e-bdf33a3d44a9 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 13 15:53:47.139768 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:53:47.139768 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:53:47.139768 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:53:47.139768 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:53:47.139768 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:53:47.139768 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:53:47.139768 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:53:47.139768 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:53:47.139768 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:53:47.139768 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:53:47.139768 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:53:47.139768 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 13 15:53:47.139768 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 13 15:53:47.139768 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 13 15:53:47.139768 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 13 15:53:47.140679 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 13 15:53:47.140679 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 13 15:53:47.140679 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 13 15:53:47.140679 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 13 15:53:47.140679 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:53:47.140679 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:53:47.140679 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:53:47.140679 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:53:47.140679 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:53:47.140679 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:53:47.140679 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:53:47.140679 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application with Lock( Sep 13 15:53:47.140679 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:53:47.140679 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:53:47.140679 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:53:47.140679 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:53:47.141479 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:53:47.141479 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:53:47.141479 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:53:47.141479 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:53:47.141479 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 13 15:53:47.141479 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 13 15:53:47.141479 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 13 15:53:47.141479 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 13 15:53:47.141479 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 13 15:53:47.141479 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 13 15:53:47.141479 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 13 15:53:47.141479 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Sep 13 15:53:47.141479 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 199/399] 10.222.0.50 () {56 vars in 1114 bytes} [Tue Sep 13 15:53:46 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 199 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:53:47.146707 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-438f3bdb-6913-4d57-98a7-0408db3e3129 None None] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:53:47.194590 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bd11c77-ead4-4165-967e-bdf33a3d44a9 None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:53:47.276880 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:53:47.300276 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-438f3bdb-6913-4d57-98a7-0408db3e3129 None None] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:53:47.340743 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-182dd692-7c8b-4eed-801f-ff47e19602c6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:53:47.341355 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-182dd692-7c8b-4eed-801f-ff47e19602c6 None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:47.341630 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-182dd692-7c8b-4eed-801f-ff47e19602c6 None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:47.341897 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-182dd692-7c8b-4eed-801f-ff47e19602c6 None admin] PATH_INFO: `/v3/roles` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:47.342554 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-182dd692-7c8b-4eed-801f-ff47e19602c6 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81437) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:53:47.342857 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-182dd692-7c8b-4eed-801f-ff47e19602c6 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:53:47.344451 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-182dd692-7c8b-4eed-801f-ff47e19602c6 None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:53:47.355535 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 201/400] 10.222.0.50 () {56 vars in 1111 bytes} [Tue Sep 13 15:53:47 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 209 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:53:47.361494 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-182dd692-7c8b-4eed-801f-ff47e19602c6 None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:53:47.414135 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bd11c77-ead4-4165-967e-bdf33a3d44a9 None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:53:47.485091 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24035509-d4a9-45ce-a3c4-9e8e946ca2d6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:53:47.486108 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24035509-d4a9-45ce-a3c4-9e8e946ca2d6 None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:47.486604 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24035509-d4a9-45ce-a3c4-9e8e946ca2d6 None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:47.487119 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24035509-d4a9-45ce-a3c4-9e8e946ca2d6 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:47.506122 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24035509-d4a9-45ce-a3c4-9e8e946ca2d6 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 13 15:53:47.512010 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-182dd692-7c8b-4eed-801f-ff47e19602c6 None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:53:47.513631 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application [None req-24035509-d4a9-45ce-a3c4-9e8e946ca2d6 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 13 15:53:47.513631 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:53:47.513631 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:53:47.513631 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:53:47.513631 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:53:47.513631 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:53:47.513631 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:53:47.513631 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:53:47.513631 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:53:47.513631 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:53:47.513631 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:53:47.513631 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:53:47.513631 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 13 15:53:47.513631 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 13 15:53:47.513631 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 13 15:53:47.513631 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 13 15:53:47.514722 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 13 15:53:47.514722 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 13 15:53:47.514722 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 13 15:53:47.514722 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 13 15:53:47.514722 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:53:47.514722 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:53:47.514722 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:53:47.514722 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:53:47.514722 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:53:47.514722 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:53:47.514722 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:53:47.514722 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application with Lock( Sep 13 15:53:47.514722 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:53:47.514722 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:53:47.514722 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:53:47.514722 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:53:47.515767 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:53:47.515767 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:53:47.515767 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:53:47.515767 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:53:47.515767 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 13 15:53:47.515767 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 13 15:53:47.515767 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 13 15:53:47.515767 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 13 15:53:47.515767 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 13 15:53:47.515767 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 13 15:53:47.515767 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 13 15:53:47.515767 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Sep 13 15:53:47.518997 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 200/401] 10.222.0.50 () {56 vars in 1099 bytes} [Tue Sep 13 15:53:47 2022] GET /identity/v3/roles/member => generated 84 bytes in 325 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:53:47.520447 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4594afc8-8d1d-4b49-a9b3-1bbe851c290d None None] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:47.520933 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4594afc8-8d1d-4b49-a9b3-1bbe851c290d None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:47.521405 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4594afc8-8d1d-4b49-a9b3-1bbe851c290d None None] PATH_INFO: `/` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:47.522847 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 201/402] 10.222.0.50 () {54 vars in 927 bytes} [Tue Sep 13 15:53:47 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:53:47.528518 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4594afc8-8d1d-4b49-a9b3-1bbe851c290d None None] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:53:47.540263 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:53:47.559292 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f816d36c-b253-4f28-bcf6-ea405de3863e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:53:47.559836 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f816d36c-b253-4f28-bcf6-ea405de3863e None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:47.560113 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f816d36c-b253-4f28-bcf6-ea405de3863e None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:47.560384 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f816d36c-b253-4f28-bcf6-ea405de3863e None admin] PATH_INFO: `/v3/groups/acdd1f9701134c2e9b5d957895b30027` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:47.561600 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f816d36c-b253-4f28-bcf6-ea405de3863e None admin] RBAC: Authorizing `identity:get_group(group_id=acdd1f9701134c2e9b5d957895b30027)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:53:47.563192 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f816d36c-b253-4f28-bcf6-ea405de3863e None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:53:47.565195 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 202/403] 10.222.0.50 () {56 vars in 1180 bytes} [Tue Sep 13 15:53:47 2022] GET /identity/v3/groups/acdd1f9701134c2e9b5d957895b30027 => generated 226 bytes in 204 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 13 15:53:47.566476 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab3fb9ee-f907-4789-925f-cfe1a6637f2f None None] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:47.566761 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab3fb9ee-f907-4789-925f-cfe1a6637f2f None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:47.567062 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab3fb9ee-f907-4789-925f-cfe1a6637f2f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:47.585979 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.auth.core [None req-ab3fb9ee-f907-4789-925f-cfe1a6637f2f None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81437) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:53:47.587191 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-ab3fb9ee-f907-4789-925f-cfe1a6637f2f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:53:47.590892 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:53:47.655889 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4594afc8-8d1d-4b49-a9b3-1bbe851c290d None None] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:53:47.670923 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 203/404] 10.222.0.50 () {58 vars in 1024 bytes} [Tue Sep 13 15:53:47 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 13 15:53:47.670923 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab3fb9ee-f907-4789-925f-cfe1a6637f2f None None] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:53:47.743636 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93be122e-3bab-4e1d-a1ce-dd3d576cfa8e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:53:47.744271 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93be122e-3bab-4e1d-a1ce-dd3d576cfa8e None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:47.744608 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93be122e-3bab-4e1d-a1ce-dd3d576cfa8e None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:47.744879 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93be122e-3bab-4e1d-a1ce-dd3d576cfa8e None admin] PATH_INFO: `/v3/roles` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:47.745559 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93be122e-3bab-4e1d-a1ce-dd3d576cfa8e None admin] RBAC: Adding query filter params (name=member) {{(pid=81438) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:53:47.745863 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93be122e-3bab-4e1d-a1ce-dd3d576cfa8e None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:53:47.747274 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93be122e-3bab-4e1d-a1ce-dd3d576cfa8e None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:53:47.766802 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 202/405] 10.222.0.50 () {56 vars in 1101 bytes} [Tue Sep 13 15:53:47 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 238 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:53:47.781973 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93be122e-3bab-4e1d-a1ce-dd3d576cfa8e None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:53:47.783997 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab3fb9ee-f907-4789-925f-cfe1a6637f2f None None] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:53:47.821506 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36b41672-7099-4e89-9fe7-2313f6ecd880 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:53:47.822065 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36b41672-7099-4e89-9fe7-2313f6ecd880 None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:47.822346 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36b41672-7099-4e89-9fe7-2313f6ecd880 None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:47.823171 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36b41672-7099-4e89-9fe7-2313f6ecd880 None admin] PATH_INFO: `/v3/projects/9ec1adca55204df2b653531664b60123` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:47.824361 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36b41672-7099-4e89-9fe7-2313f6ecd880 None admin] RBAC: Authorizing `identity:get_project(project_id=9ec1adca55204df2b653531664b60123)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:53:47.825865 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36b41672-7099-4e89-9fe7-2313f6ecd880 None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:53:47.827880 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 204/406] 10.222.0.50 () {56 vars in 1186 bytes} [Tue Sep 13 15:53:47 2022] GET /identity/v3/projects/9ec1adca55204df2b653531664b60123 => generated 302 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:53:47.833463 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36b41672-7099-4e89-9fe7-2313f6ecd880 None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:53:47.854397 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:53:47.935114 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:53:48.002881 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36b41672-7099-4e89-9fe7-2313f6ecd880 None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:53:48.005529 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93be122e-3bab-4e1d-a1ce-dd3d576cfa8e None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:53:48.040248 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee611bbf-3806-4d24-976c-2ab898b06957 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:53:48.041830 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee611bbf-3806-4d24-976c-2ab898b06957 None admin] REQUEST_METHOD: `PUT` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:48.042374 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee611bbf-3806-4d24-976c-2ab898b06957 None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:48.042851 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee611bbf-3806-4d24-976c-2ab898b06957 None admin] PATH_INFO: `/v3/projects/9ec1adca55204df2b653531664b60123/groups/acdd1f9701134c2e9b5d957895b30027/roles/f74e3f6c20bb426ca030f211c3a9aef5` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:48.045500 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee611bbf-3806-4d24-976c-2ab898b06957 None admin] RBAC: Authorizing `identity:create_grant(project_id=9ec1adca55204df2b653531664b60123, group_id=acdd1f9701134c2e9b5d957895b30027, role_id=f74e3f6c20bb426ca030f211c3a9aef5)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:53:48.047290 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee611bbf-3806-4d24-976c-2ab898b06957 None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:53:48.054452 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9223e03-2e60-4411-a747-9dfd6efb87e7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:53:48.055187 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9223e03-2e60-4411-a747-9dfd6efb87e7 None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:48.055509 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9223e03-2e60-4411-a747-9dfd6efb87e7 None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:48.055739 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9223e03-2e60-4411-a747-9dfd6efb87e7 None admin] PATH_INFO: `/v3/groups/acdd1f9701134c2e9b5d957895b30027` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:48.058071 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9223e03-2e60-4411-a747-9dfd6efb87e7 None admin] RBAC: Authorizing `identity:get_group(group_id=acdd1f9701134c2e9b5d957895b30027)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:53:48.062735 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9223e03-2e60-4411-a747-9dfd6efb87e7 None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:53:48.066978 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 203/407] 10.222.0.50 () {56 vars in 1180 bytes} [Tue Sep 13 15:53:47 2022] GET /identity/v3/groups/acdd1f9701134c2e9b5d957895b30027 => generated 226 bytes in 283 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 13 15:53:48.068064 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9223e03-2e60-4411-a747-9dfd6efb87e7 None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:53:48.074829 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 205/408] 10.222.0.50 () {58 vars in 1442 bytes} [Tue Sep 13 15:53:47 2022] PUT /identity/v3/projects/9ec1adca55204df2b653531664b60123/groups/acdd1f9701134c2e9b5d957895b30027/roles/f74e3f6c20bb426ca030f211c3a9aef5 => generated 0 bytes in 241 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:53:48.076124 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee611bbf-3806-4d24-976c-2ab898b06957 None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:53:48.162637 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:53:48.190449 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9223e03-2e60-4411-a747-9dfd6efb87e7 None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:53:48.203497 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee611bbf-3806-4d24-976c-2ab898b06957 None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:53:48.230213 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d22bc63b-b387-4896-af7e-aae97d0884d1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:53:48.231306 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d22bc63b-b387-4896-af7e-aae97d0884d1 None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:48.231795 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d22bc63b-b387-4896-af7e-aae97d0884d1 None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:48.232279 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d22bc63b-b387-4896-af7e-aae97d0884d1 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:48.237459 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:53:48.243498 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b3ebff2-0c4b-4eab-bc74-560e680d9335 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:53:48.244264 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b3ebff2-0c4b-4eab-bc74-560e680d9335 None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:48.245049 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b3ebff2-0c4b-4eab-bc74-560e680d9335 None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:48.245463 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b3ebff2-0c4b-4eab-bc74-560e680d9335 None admin] PATH_INFO: `/v3/projects/9ec1adca55204df2b653531664b60123` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:48.246736 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b3ebff2-0c4b-4eab-bc74-560e680d9335 None admin] RBAC: Authorizing `identity:get_project(project_id=9ec1adca55204df2b653531664b60123)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:53:48.248290 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b3ebff2-0c4b-4eab-bc74-560e680d9335 None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:53:48.249728 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d22bc63b-b387-4896-af7e-aae97d0884d1 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 13 15:53:48.254886 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 206/409] 10.222.0.50 () {56 vars in 1186 bytes} [Tue Sep 13 15:53:48 2022] GET /identity/v3/projects/9ec1adca55204df2b653531664b60123 => generated 302 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:53:48.255290 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application [None req-d22bc63b-b387-4896-af7e-aae97d0884d1 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 13 15:53:48.255290 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:53:48.255290 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:53:48.255290 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:53:48.255290 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:53:48.255290 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:53:48.255290 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:53:48.255290 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:53:48.255290 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:53:48.255290 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:53:48.255290 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:53:48.255290 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:53:48.255290 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 13 15:53:48.255290 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 13 15:53:48.255290 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 13 15:53:48.255290 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 13 15:53:48.256263 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 13 15:53:48.256263 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 13 15:53:48.256263 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 13 15:53:48.256263 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 13 15:53:48.256263 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:53:48.256263 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:53:48.256263 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:53:48.256263 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:53:48.256263 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:53:48.256263 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:53:48.256263 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:53:48.256263 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application with Lock( Sep 13 15:53:48.256263 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:53:48.256263 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:53:48.256263 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:53:48.256263 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:53:48.257075 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:53:48.257075 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:53:48.257075 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:53:48.257075 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:53:48.257075 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 13 15:53:48.257075 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 13 15:53:48.257075 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 13 15:53:48.257075 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 13 15:53:48.257075 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 13 15:53:48.257075 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 13 15:53:48.257075 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 13 15:53:48.257075 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Sep 13 15:53:48.257746 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b3ebff2-0c4b-4eab-bc74-560e680d9335 None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:53:48.259373 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 204/410] 10.222.0.50 () {56 vars in 1096 bytes} [Tue Sep 13 15:53:48 2022] GET /identity/v3/roles/admin => generated 83 bytes in 192 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:53:48.265805 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d22bc63b-b387-4896-af7e-aae97d0884d1 None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:53:48.380251 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d22bc63b-b387-4896-af7e-aae97d0884d1 None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:53:48.380848 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b3ebff2-0c4b-4eab-bc74-560e680d9335 None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:53:48.419236 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7d1c717-947b-492e-bba2-f89efa4dcf6a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:53:48.420375 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7d1c717-947b-492e-bba2-f89efa4dcf6a None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:48.420849 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7d1c717-947b-492e-bba2-f89efa4dcf6a None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:48.421298 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7d1c717-947b-492e-bba2-f89efa4dcf6a None admin] PATH_INFO: `/v3/roles` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:48.421477 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3191d7bd-e228-420e-8103-68070e4354c0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:53:48.422246 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3191d7bd-e228-420e-8103-68070e4354c0 None admin] REQUEST_METHOD: `PUT` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:48.422436 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7d1c717-947b-492e-bba2-f89efa4dcf6a None admin] RBAC: Adding query filter params (name=admin) {{(pid=81438) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:53:48.422712 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3191d7bd-e228-420e-8103-68070e4354c0 None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:48.423115 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3191d7bd-e228-420e-8103-68070e4354c0 None admin] PATH_INFO: `/v3/projects/9ec1adca55204df2b653531664b60123/groups/acdd1f9701134c2e9b5d957895b30027/roles/15db4aaca3264ae29bde38903ba43471` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:48.423287 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7d1c717-947b-492e-bba2-f89efa4dcf6a None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:53:48.424811 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7d1c717-947b-492e-bba2-f89efa4dcf6a None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:53:48.425334 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3191d7bd-e228-420e-8103-68070e4354c0 None admin] RBAC: Authorizing `identity:create_grant(project_id=9ec1adca55204df2b653531664b60123, group_id=acdd1f9701134c2e9b5d957895b30027, role_id=15db4aaca3264ae29bde38903ba43471)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:53:48.427276 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3191d7bd-e228-420e-8103-68070e4354c0 None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:53:48.432341 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:53:48.435989 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 205/411] 10.222.0.50 () {56 vars in 1099 bytes} [Tue Sep 13 15:53:48 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 13 15:53:48.437386 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f6d9ac0-47e6-4930-bda0-4ecc2c1d605a None None] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:48.437624 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f6d9ac0-47e6-4930-bda0-4ecc2c1d605a None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:48.437906 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f6d9ac0-47e6-4930-bda0-4ecc2c1d605a None None] PATH_INFO: `/` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:48.439479 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 206/412] 10.222.0.50 () {54 vars in 927 bytes} [Tue Sep 13 15:53:48 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:53:48.443693 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f6d9ac0-47e6-4930-bda0-4ecc2c1d605a None None] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:53:48.449525 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 207/413] 10.222.0.50 () {58 vars in 1442 bytes} [Tue Sep 13 15:53:48 2022] PUT /identity/v3/projects/9ec1adca55204df2b653531664b60123/groups/acdd1f9701134c2e9b5d957895b30027/roles/15db4aaca3264ae29bde38903ba43471 => generated 0 bytes in 194 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:53:48.451223 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c505b28b-ce63-4e9e-bb58-485728d44a83 None None] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:48.451751 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c505b28b-ce63-4e9e-bb58-485728d44a83 None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:48.452226 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c505b28b-ce63-4e9e-bb58-485728d44a83 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:48.472941 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.auth.core [None req-c505b28b-ce63-4e9e-bb58-485728d44a83 None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81437) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:53:48.474541 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-c505b28b-ce63-4e9e-bb58-485728d44a83 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:53:48.518685 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:53:48.587976 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f6d9ac0-47e6-4930-bda0-4ecc2c1d605a None None] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:53:48.596376 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 208/414] 10.222.0.50 () {58 vars in 1024 bytes} [Tue Sep 13 15:53:48 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 146 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 13 15:53:48.638378 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ee6cd41-7f27-4477-b72e-8a3c3c5f764c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:53:48.638983 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ee6cd41-7f27-4477-b72e-8a3c3c5f764c None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:48.639335 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ee6cd41-7f27-4477-b72e-8a3c3c5f764c None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:48.639621 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ee6cd41-7f27-4477-b72e-8a3c3c5f764c None admin] PATH_INFO: `/v3/groups/58a500e5c4724a8eb4916f4251241715` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:48.640894 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ee6cd41-7f27-4477-b72e-8a3c3c5f764c None admin] RBAC: Authorizing `identity:get_group(group_id=58a500e5c4724a8eb4916f4251241715)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:53:48.642396 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ee6cd41-7f27-4477-b72e-8a3c3c5f764c None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:53:48.644016 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 207/415] 10.222.0.50 () {56 vars in 1180 bytes} [Tue Sep 13 15:53:48 2022] GET /identity/v3/groups/58a500e5c4724a8eb4916f4251241715 => generated 229 bytes in 201 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:53:48.649770 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c505b28b-ce63-4e9e-bb58-485728d44a83 None None] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:53:48.766471 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c505b28b-ce63-4e9e-bb58-485728d44a83 None None] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:53:48.795029 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:53:48.799470 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-435d9637-0ab2-46c1-812f-3a2808d44f7a None None] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:48.800149 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-435d9637-0ab2-46c1-812f-3a2808d44f7a None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:48.800425 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-435d9637-0ab2-46c1-812f-3a2808d44f7a None None] PATH_INFO: `/` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:48.803437 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 208/416] 10.222.0.50 () {54 vars in 927 bytes} [Tue Sep 13 15:53:48 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:53:48.808575 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce1720d6-8d4c-4f25-9269-0af53b2a352c None None] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:48.808839 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce1720d6-8d4c-4f25-9269-0af53b2a352c None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:48.809052 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce1720d6-8d4c-4f25-9269-0af53b2a352c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:48.812647 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0baa12f8-9ff2-46ac-8646-3f41125a1626 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:53:48.813305 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0baa12f8-9ff2-46ac-8646-3f41125a1626 None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:48.813582 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0baa12f8-9ff2-46ac-8646-3f41125a1626 None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:48.813856 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0baa12f8-9ff2-46ac-8646-3f41125a1626 None admin] PATH_INFO: `/v3/projects/555bee08a3fa4d08bdd8ecc881adc45b` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:48.815018 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0baa12f8-9ff2-46ac-8646-3f41125a1626 None admin] RBAC: Authorizing `identity:get_project(project_id=555bee08a3fa4d08bdd8ecc881adc45b)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:53:48.816472 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0baa12f8-9ff2-46ac-8646-3f41125a1626 None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:53:48.818106 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 209/417] 10.222.0.50 () {56 vars in 1186 bytes} [Tue Sep 13 15:53:48 2022] GET /identity/v3/projects/555bee08a3fa4d08bdd8ecc881adc45b => generated 344 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:53:48.824252 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0baa12f8-9ff2-46ac-8646-3f41125a1626 None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:53:48.830145 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.auth.core [None req-ce1720d6-8d4c-4f25-9269-0af53b2a352c None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81438) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:53:48.831669 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-ce1720d6-8d4c-4f25-9269-0af53b2a352c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:53:48.914882 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 209/418] 10.222.0.50 () {58 vars in 1024 bytes} [Tue Sep 13 15:53:48 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 13 15:53:48.959988 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce1720d6-8d4c-4f25-9269-0af53b2a352c None None] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:53:49.008844 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0baa12f8-9ff2-46ac-8646-3f41125a1626 None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:53:49.066390 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:53:49.078515 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fd8d743-da4a-4515-ba8e-a62ee165226f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:53:49.079624 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fd8d743-da4a-4515-ba8e-a62ee165226f None admin] REQUEST_METHOD: `PUT` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:49.080187 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fd8d743-da4a-4515-ba8e-a62ee165226f None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:49.080663 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fd8d743-da4a-4515-ba8e-a62ee165226f None admin] PATH_INFO: `/v3/projects/555bee08a3fa4d08bdd8ecc881adc45b/groups/58a500e5c4724a8eb4916f4251241715/roles/031d4e1a23e34f71858fc4a6dfbd4ccb` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:49.091263 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fd8d743-da4a-4515-ba8e-a62ee165226f None admin] RBAC: Authorizing `identity:create_grant(project_id=555bee08a3fa4d08bdd8ecc881adc45b, group_id=58a500e5c4724a8eb4916f4251241715, role_id=031d4e1a23e34f71858fc4a6dfbd4ccb)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:53:49.093092 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fd8d743-da4a-4515-ba8e-a62ee165226f None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:53:49.117863 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 210/419] 10.222.0.50 () {58 vars in 1442 bytes} [Tue Sep 13 15:53:48 2022] PUT /identity/v3/projects/555bee08a3fa4d08bdd8ecc881adc45b/groups/58a500e5c4724a8eb4916f4251241715/roles/031d4e1a23e34f71858fc4a6dfbd4ccb => generated 0 bytes in 294 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 13 15:53:49.119925 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce1720d6-8d4c-4f25-9269-0af53b2a352c None None] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:53:49.158306 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37c4d59b-14bf-41c7-82e9-521954bebcf2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:53:49.158991 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37c4d59b-14bf-41c7-82e9-521954bebcf2 None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:49.159271 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37c4d59b-14bf-41c7-82e9-521954bebcf2 None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:49.159542 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37c4d59b-14bf-41c7-82e9-521954bebcf2 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:49.171526 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37c4d59b-14bf-41c7-82e9-521954bebcf2 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 13 15:53:49.176331 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application [None req-37c4d59b-14bf-41c7-82e9-521954bebcf2 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 13 15:53:49.176331 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:53:49.176331 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:53:49.176331 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:53:49.176331 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:53:49.176331 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:53:49.176331 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:53:49.176331 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:53:49.176331 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:53:49.176331 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:53:49.176331 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:53:49.176331 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:53:49.176331 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 13 15:53:49.176331 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 13 15:53:49.176331 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 13 15:53:49.176331 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 13 15:53:49.177262 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 13 15:53:49.177262 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 13 15:53:49.177262 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 13 15:53:49.177262 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 13 15:53:49.177262 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:53:49.177262 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:53:49.177262 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:53:49.177262 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:53:49.177262 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:53:49.177262 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:53:49.177262 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:53:49.177262 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application with Lock( Sep 13 15:53:49.177262 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:53:49.177262 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:53:49.177262 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:53:49.177262 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:53:49.178059 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:53:49.178059 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:53:49.178059 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:53:49.178059 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:53:49.178059 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 13 15:53:49.178059 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 13 15:53:49.178059 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 13 15:53:49.178059 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 13 15:53:49.178059 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 13 15:53:49.178059 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 13 15:53:49.178059 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 13 15:53:49.178059 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Sep 13 15:53:49.178059 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 210/420] 10.222.0.50 () {56 vars in 1099 bytes} [Tue Sep 13 15:53:48 2022] GET /identity/v3/roles/member => generated 84 bytes in 218 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 13 15:53:49.186109 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fd8d743-da4a-4515-ba8e-a62ee165226f None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:53:49.217320 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37c4d59b-14bf-41c7-82e9-521954bebcf2 None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:53:49.339283 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:53:49.370950 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37c4d59b-14bf-41c7-82e9-521954bebcf2 None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:53:49.392281 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:53:49.401434 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fd8d743-da4a-4515-ba8e-a62ee165226f None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:53:49.416891 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70d44525-8008-4796-83a0-3560a437e29a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:53:49.417684 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70d44525-8008-4796-83a0-3560a437e29a None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:49.417960 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70d44525-8008-4796-83a0-3560a437e29a None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:49.418225 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70d44525-8008-4796-83a0-3560a437e29a None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:49.431566 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70d44525-8008-4796-83a0-3560a437e29a None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 13 15:53:49.437999 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application [None req-70d44525-8008-4796-83a0-3560a437e29a None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 13 15:53:49.437999 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:53:49.437999 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:53:49.437999 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:53:49.437999 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:53:49.437999 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:53:49.437999 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:53:49.437999 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:53:49.437999 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:53:49.437999 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:53:49.437999 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:53:49.437999 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:53:49.437999 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 13 15:53:49.437999 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 13 15:53:49.437999 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 13 15:53:49.437999 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 13 15:53:49.439179 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 13 15:53:49.439179 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 13 15:53:49.439179 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 13 15:53:49.439179 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 13 15:53:49.439179 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:53:49.439179 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:53:49.439179 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:53:49.439179 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:53:49.439179 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:53:49.439179 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:53:49.439179 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:53:49.439179 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application with Lock( Sep 13 15:53:49.439179 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:53:49.439179 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:53:49.439179 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:53:49.439179 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:53:49.443750 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:53:49.443750 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:53:49.443750 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:53:49.443750 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:53:49.443750 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 13 15:53:49.443750 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 13 15:53:49.443750 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 13 15:53:49.443750 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 13 15:53:49.443750 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 13 15:53:49.443750 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 13 15:53:49.443750 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 13 15:53:49.443750 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Sep 13 15:53:49.443750 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 211/421] 10.222.0.50 () {56 vars in 1114 bytes} [Tue Sep 13 15:53:49 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 223 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 13 15:53:49.446105 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49250403-8067-4913-ba26-a517c02831c5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:53:49.446573 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49250403-8067-4913-ba26-a517c02831c5 None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:49.446849 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49250403-8067-4913-ba26-a517c02831c5 None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:49.447142 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49250403-8067-4913-ba26-a517c02831c5 None admin] PATH_INFO: `/v3/roles` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:49.448349 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49250403-8067-4913-ba26-a517c02831c5 None admin] RBAC: Adding query filter params (name=member) {{(pid=81437) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:53:49.448631 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49250403-8067-4913-ba26-a517c02831c5 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:53:49.450018 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49250403-8067-4913-ba26-a517c02831c5 None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:53:49.451894 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70d44525-8008-4796-83a0-3560a437e29a None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:53:49.478436 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 211/422] 10.222.0.50 () {56 vars in 1101 bytes} [Tue Sep 13 15:53:49 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 293 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:53:49.484506 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49250403-8067-4913-ba26-a517c02831c5 None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:53:49.629036 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70d44525-8008-4796-83a0-3560a437e29a None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:53:49.666633 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80c5b616-29c1-4ed2-badd-ba452b006dd4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:53:49.667406 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80c5b616-29c1-4ed2-badd-ba452b006dd4 None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:49.667682 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80c5b616-29c1-4ed2-badd-ba452b006dd4 None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:49.667954 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80c5b616-29c1-4ed2-badd-ba452b006dd4 None admin] PATH_INFO: `/v3/roles` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:49.669029 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80c5b616-29c1-4ed2-badd-ba452b006dd4 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81438) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:53:49.669332 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80c5b616-29c1-4ed2-badd-ba452b006dd4 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:53:49.670697 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80c5b616-29c1-4ed2-badd-ba452b006dd4 None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:53:49.679863 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 212/423] 10.222.0.50 () {56 vars in 1111 bytes} [Tue Sep 13 15:53:49 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 228 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:53:49.686079 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80c5b616-29c1-4ed2-badd-ba452b006dd4 None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:53:49.733396 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49250403-8067-4913-ba26-a517c02831c5 None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:53:49.735099 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:53:49.772773 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:53:49.814009 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a94f38f-8446-4638-9478-c758a929d971 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:53:49.814629 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a94f38f-8446-4638-9478-c758a929d971 None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:49.814921 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a94f38f-8446-4638-9478-c758a929d971 None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:49.815196 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a94f38f-8446-4638-9478-c758a929d971 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:49.817492 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80c5b616-29c1-4ed2-badd-ba452b006dd4 None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:53:49.900090 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a94f38f-8446-4638-9478-c758a929d971 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:53:49.901615 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a94f38f-8446-4638-9478-c758a929d971 None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:53:49.904253 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ffc9777-d5f8-41a5-9115-7e794231f170 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:53:49.905157 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ffc9777-d5f8-41a5-9115-7e794231f170 None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:49.905638 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ffc9777-d5f8-41a5-9115-7e794231f170 None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:49.906147 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ffc9777-d5f8-41a5-9115-7e794231f170 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:50.026605 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ffc9777-d5f8-41a5-9115-7e794231f170 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:53:50.029201 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ffc9777-d5f8-41a5-9115-7e794231f170 None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:53:50.055481 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 212/424] 10.222.0.50 () {58 vars in 1282 bytes} [Tue Sep 13 15:53:49 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 571 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 13 15:53:50.062332 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a94f38f-8446-4638-9478-c758a929d971 None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:53:50.091341 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:53:50.119369 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:53:50.171224 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 213/425] 10.222.0.50 () {58 vars in 1282 bytes} [Tue Sep 13 15:53:49 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 485 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 13 15:53:50.178164 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ffc9777-d5f8-41a5-9115-7e794231f170 None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:53:50.299099 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ffc9777-d5f8-41a5-9115-7e794231f170 None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:53:50.314069 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a94f38f-8446-4638-9478-c758a929d971 None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:53:50.367786 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-752ec9b7-4c90-450b-bd7e-8504694db1cf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:53:50.368790 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-752ec9b7-4c90-450b-bd7e-8504694db1cf None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:50.369273 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-752ec9b7-4c90-450b-bd7e-8504694db1cf None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:50.369958 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-752ec9b7-4c90-450b-bd7e-8504694db1cf None admin] PATH_INFO: `/v3/projects/423edf83b2b24540924e990bb324af1e` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:50.371495 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-752ec9b7-4c90-450b-bd7e-8504694db1cf None admin] RBAC: Authorizing `identity:get_project(project_id=423edf83b2b24540924e990bb324af1e)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:53:50.374012 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-752ec9b7-4c90-450b-bd7e-8504694db1cf None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:53:50.378420 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 214/426] 10.222.0.50 () {56 vars in 1186 bytes} [Tue Sep 13 15:53:50 2022] GET /identity/v3/projects/423edf83b2b24540924e990bb324af1e => generated 298 bytes in 199 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:53:50.379300 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcc99974-21c6-4a57-800a-d6c31b805264 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:53:50.380189 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcc99974-21c6-4a57-800a-d6c31b805264 None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:50.380701 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcc99974-21c6-4a57-800a-d6c31b805264 None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:50.381705 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-752ec9b7-4c90-450b-bd7e-8504694db1cf None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:53:50.382188 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcc99974-21c6-4a57-800a-d6c31b805264 None admin] PATH_INFO: `/v3/projects/423edf83b2b24540924e990bb324af1e` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:50.383886 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dcc99974-21c6-4a57-800a-d6c31b805264 None admin] RBAC: Authorizing `identity:get_project(project_id=423edf83b2b24540924e990bb324af1e)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:53:50.386436 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dcc99974-21c6-4a57-800a-d6c31b805264 None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:53:50.390182 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 213/427] 10.222.0.50 () {56 vars in 1186 bytes} [Tue Sep 13 15:53:50 2022] GET /identity/v3/projects/423edf83b2b24540924e990bb324af1e => generated 298 bytes in 329 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 13 15:53:50.396437 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcc99974-21c6-4a57-800a-d6c31b805264 None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:53:50.407262 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:53:50.564194 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:53:50.581219 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-752ec9b7-4c90-450b-bd7e-8504694db1cf None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:53:50.597871 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcc99974-21c6-4a57-800a-d6c31b805264 None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:53:50.653507 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07ff13d5-8756-4eb0-a87d-aef5009d5370 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:53:50.654147 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07ff13d5-8756-4eb0-a87d-aef5009d5370 None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:50.654424 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07ff13d5-8756-4eb0-a87d-aef5009d5370 None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:50.654693 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07ff13d5-8756-4eb0-a87d-aef5009d5370 None admin] PATH_INFO: `/v3/groups/acdd1f9701134c2e9b5d957895b30027` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:50.655820 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bea81d8e-055f-4e25-967a-7ddc25394aa2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:53:50.656503 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bea81d8e-055f-4e25-967a-7ddc25394aa2 None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:50.656777 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bea81d8e-055f-4e25-967a-7ddc25394aa2 None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:50.657045 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bea81d8e-055f-4e25-967a-7ddc25394aa2 None admin] PATH_INFO: `/v3/groups/acdd1f9701134c2e9b5d957895b30027` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:50.658113 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07ff13d5-8756-4eb0-a87d-aef5009d5370 None admin] RBAC: Authorizing `identity:get_group(group_id=acdd1f9701134c2e9b5d957895b30027)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:53:50.659575 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07ff13d5-8756-4eb0-a87d-aef5009d5370 None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:53:50.659835 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bea81d8e-055f-4e25-967a-7ddc25394aa2 None admin] RBAC: Authorizing `identity:get_group(group_id=acdd1f9701134c2e9b5d957895b30027)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:53:50.661438 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 214/428] 10.222.0.50 () {56 vars in 1180 bytes} [Tue Sep 13 15:53:50 2022] GET /identity/v3/groups/acdd1f9701134c2e9b5d957895b30027 => generated 226 bytes in 265 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:53:50.661714 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bea81d8e-055f-4e25-967a-7ddc25394aa2 None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:53:50.670948 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 215/429] 10.222.0.50 () {56 vars in 1180 bytes} [Tue Sep 13 15:53:50 2022] GET /identity/v3/groups/acdd1f9701134c2e9b5d957895b30027 => generated 226 bytes in 282 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:53:50.672123 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07ff13d5-8756-4eb0-a87d-aef5009d5370 None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:53:50.673173 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bea81d8e-055f-4e25-967a-7ddc25394aa2 None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:53:50.833383 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:53:50.861360 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07ff13d5-8756-4eb0-a87d-aef5009d5370 None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:53:50.870950 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bea81d8e-055f-4e25-967a-7ddc25394aa2 None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:53:50.901624 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57eb11a4-022c-4a41-bebe-30e07b289919 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:53:50.902238 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57eb11a4-022c-4a41-bebe-30e07b289919 None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:50.902626 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57eb11a4-022c-4a41-bebe-30e07b289919 None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:50.902937 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57eb11a4-022c-4a41-bebe-30e07b289919 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:50.903764 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57eb11a4-022c-4a41-bebe-30e07b289919 None admin] RBAC: Adding query filter params (group.id=acdd1f9701134c2e9b5d957895b30027, role.id=15db4aaca3264ae29bde38903ba43471, scope.project.id=423edf83b2b24540924e990bb324af1e) {{(pid=81437) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:53:50.904122 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57eb11a4-022c-4a41-bebe-30e07b289919 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:53:50.905665 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57eb11a4-022c-4a41-bebe-30e07b289919 None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:53:50.910960 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79c94998-cf00-421e-9903-94f4a1b09c9e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:53:50.911561 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79c94998-cf00-421e-9903-94f4a1b09c9e None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:50.911836 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79c94998-cf00-421e-9903-94f4a1b09c9e None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:50.912103 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79c94998-cf00-421e-9903-94f4a1b09c9e None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:50.912220 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 215/430] 10.222.0.50 () {56 vars in 1376 bytes} [Tue Sep 13 15:53:50 2022] GET /identity/v3/role_assignments?group.id=acdd1f9701134c2e9b5d957895b30027&scope.project.id=423edf83b2b24540924e990bb324af1e&role.id=15db4aaca3264ae29bde38903ba43471 => generated 611 bytes in 240 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:53:50.912942 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79c94998-cf00-421e-9903-94f4a1b09c9e None admin] RBAC: Adding query filter params (group.id=acdd1f9701134c2e9b5d957895b30027, role.id=f74e3f6c20bb426ca030f211c3a9aef5, scope.project.id=423edf83b2b24540924e990bb324af1e) {{(pid=81438) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:53:50.913250 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79c94998-cf00-421e-9903-94f4a1b09c9e None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:53:50.913525 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-088afc29-daca-43b0-ba8d-0f8546828dcb None None] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:50.913813 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-088afc29-daca-43b0-ba8d-0f8546828dcb None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:50.914152 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-088afc29-daca-43b0-ba8d-0f8546828dcb None None] PATH_INFO: `/` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:50.914685 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79c94998-cf00-421e-9903-94f4a1b09c9e None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:53:50.920552 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 216/431] 10.222.0.50 () {54 vars in 927 bytes} [Tue Sep 13 15:53:50 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:53:50.922165 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e60240f3-900f-4c18-a95e-94e4d5aba09f None None] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:50.922572 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e60240f3-900f-4c18-a95e-94e4d5aba09f None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:50.922831 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 216/432] 10.222.0.50 () {56 vars in 1376 bytes} [Tue Sep 13 15:53:50 2022] GET /identity/v3/role_assignments?group.id=acdd1f9701134c2e9b5d957895b30027&scope.project.id=423edf83b2b24540924e990bb324af1e&role.id=f74e3f6c20bb426ca030f211c3a9aef5 => generated 611 bytes in 250 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:53:50.922953 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e60240f3-900f-4c18-a95e-94e4d5aba09f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:50.950338 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.auth.core [None req-e60240f3-900f-4c18-a95e-94e4d5aba09f None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81437) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:53:50.955958 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-e60240f3-900f-4c18-a95e-94e4d5aba09f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:53:50.975426 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:53:51.068611 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 217/433] 10.222.0.50 () {58 vars in 1024 bytes} [Tue Sep 13 15:53:50 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 147 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 13 15:53:51.091829 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85a1c564-1fe7-4666-b3a8-e14a1b9a9e13 None None] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:51.092433 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85a1c564-1fe7-4666-b3a8-e14a1b9a9e13 None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:51.092916 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85a1c564-1fe7-4666-b3a8-e14a1b9a9e13 None None] PATH_INFO: `/` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:51.094525 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 217/434] 10.222.0.50 () {54 vars in 927 bytes} [Tue Sep 13 15:53:51 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:53:51.100615 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68c28ac2-1fb4-45f6-aa7b-c0eeb8339001 None None] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:51.100969 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68c28ac2-1fb4-45f6-aa7b-c0eeb8339001 None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:51.101248 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68c28ac2-1fb4-45f6-aa7b-c0eeb8339001 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:51.127357 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.auth.core [None req-68c28ac2-1fb4-45f6-aa7b-c0eeb8339001 None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81437) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:53:51.128547 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-68c28ac2-1fb4-45f6-aa7b-c0eeb8339001 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:53:51.227413 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 218/435] 10.222.0.50 () {58 vars in 1024 bytes} [Tue Sep 13 15:53:51 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 128 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 13 15:53:51.374775 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85a1c564-1fe7-4666-b3a8-e14a1b9a9e13 None None] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:53:51.484731 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85a1c564-1fe7-4666-b3a8-e14a1b9a9e13 None None] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:53:51.498509 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68c28ac2-1fb4-45f6-aa7b-c0eeb8339001 None None] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:53:51.529150 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7ccb672-8fe3-42a4-9e04-93ae4fe5e89f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:53:51.529798 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7ccb672-8fe3-42a4-9e04-93ae4fe5e89f None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:51.530075 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7ccb672-8fe3-42a4-9e04-93ae4fe5e89f None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:51.530344 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7ccb672-8fe3-42a4-9e04-93ae4fe5e89f None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:51.541320 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:53:51.547477 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7ccb672-8fe3-42a4-9e04-93ae4fe5e89f None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 13 15:53:51.552956 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application [None req-c7ccb672-8fe3-42a4-9e04-93ae4fe5e89f None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 13 15:53:51.552956 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:53:51.552956 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:53:51.552956 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:53:51.552956 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:53:51.552956 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:53:51.552956 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:53:51.552956 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:53:51.552956 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:53:51.552956 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:53:51.552956 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:53:51.552956 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:53:51.552956 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 13 15:53:51.552956 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 13 15:53:51.552956 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 13 15:53:51.552956 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 13 15:53:51.554091 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 13 15:53:51.554091 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 13 15:53:51.554091 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 13 15:53:51.554091 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 13 15:53:51.554091 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:53:51.554091 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:53:51.554091 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:53:51.554091 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:53:51.554091 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:53:51.554091 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:53:51.554091 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:53:51.554091 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application with Lock( Sep 13 15:53:51.554091 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:53:51.554091 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:53:51.554091 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:53:51.554091 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:53:51.554969 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:53:51.554969 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:53:51.554969 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:53:51.554969 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:53:51.554969 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 13 15:53:51.554969 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 13 15:53:51.554969 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 13 15:53:51.554969 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 13 15:53:51.554969 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 13 15:53:51.554969 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 13 15:53:51.554969 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 13 15:53:51.554969 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Sep 13 15:53:51.554969 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 218/436] 10.222.0.50 () {56 vars in 1114 bytes} [Tue Sep 13 15:53:51 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 180 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:53:51.562893 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7ccb672-8fe3-42a4-9e04-93ae4fe5e89f None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:53:51.601610 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:53:51.633275 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68c28ac2-1fb4-45f6-aa7b-c0eeb8339001 None None] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:53:51.678976 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49e05419-29fb-4f79-96de-db643dda7f83 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:53:51.679626 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49e05419-29fb-4f79-96de-db643dda7f83 None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:51.679906 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49e05419-29fb-4f79-96de-db643dda7f83 None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:51.680183 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49e05419-29fb-4f79-96de-db643dda7f83 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:51.698368 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49e05419-29fb-4f79-96de-db643dda7f83 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 13 15:53:51.703433 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application [None req-49e05419-29fb-4f79-96de-db643dda7f83 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 13 15:53:51.703433 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:53:51.703433 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:53:51.703433 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:53:51.703433 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:53:51.703433 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:53:51.703433 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:53:51.703433 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:53:51.703433 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:53:51.703433 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:53:51.703433 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:53:51.703433 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:53:51.703433 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 13 15:53:51.703433 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 13 15:53:51.703433 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 13 15:53:51.703433 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 13 15:53:51.704457 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 13 15:53:51.704457 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 13 15:53:51.704457 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 13 15:53:51.704457 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 13 15:53:51.704457 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:53:51.704457 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:53:51.704457 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:53:51.704457 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:53:51.704457 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:53:51.704457 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:53:51.704457 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:53:51.704457 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application with Lock( Sep 13 15:53:51.704457 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:53:51.704457 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:53:51.704457 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:53:51.704457 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:53:51.705308 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:53:51.705308 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:53:51.705308 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:53:51.705308 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:53:51.705308 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 13 15:53:51.705308 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 13 15:53:51.705308 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 13 15:53:51.705308 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 13 15:53:51.705308 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 13 15:53:51.705308 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 13 15:53:51.705308 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 13 15:53:51.705308 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application Sep 13 15:53:51.705308 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 219/437] 10.222.0.50 () {56 vars in 1099 bytes} [Tue Sep 13 15:53:51 2022] GET /identity/v3/roles/member => generated 84 bytes in 207 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:53:51.711109 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49e05419-29fb-4f79-96de-db643dda7f83 None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:53:51.717325 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7ccb672-8fe3-42a4-9e04-93ae4fe5e89f None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:53:51.758208 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8967aae7-0843-49f6-af1a-9974d414265d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:53:51.758834 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8967aae7-0843-49f6-af1a-9974d414265d None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:51.759156 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8967aae7-0843-49f6-af1a-9974d414265d None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:51.759432 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8967aae7-0843-49f6-af1a-9974d414265d None admin] PATH_INFO: `/v3/roles` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:51.760669 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8967aae7-0843-49f6-af1a-9974d414265d None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81438) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:53:51.760974 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8967aae7-0843-49f6-af1a-9974d414265d None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:53:51.762486 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8967aae7-0843-49f6-af1a-9974d414265d None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:53:51.771562 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 219/438] 10.222.0.50 () {56 vars in 1111 bytes} [Tue Sep 13 15:53:51 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 211 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:53:51.772791 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87be40ab-a382-40e9-8a77-27f8b6dbd0be None None] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:51.773136 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87be40ab-a382-40e9-8a77-27f8b6dbd0be None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:51.773415 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87be40ab-a382-40e9-8a77-27f8b6dbd0be None None] PATH_INFO: `/` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:51.774730 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 220/439] 10.222.0.50 () {54 vars in 927 bytes} [Tue Sep 13 15:53:51 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:53:51.777598 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87be40ab-a382-40e9-8a77-27f8b6dbd0be None None] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:53:51.827180 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49e05419-29fb-4f79-96de-db643dda7f83 None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:53:51.839059 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:53:51.860714 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c2dafec-b9bd-41dd-b59e-cb38f2e8722d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:53:51.861320 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c2dafec-b9bd-41dd-b59e-cb38f2e8722d None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:51.861598 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c2dafec-b9bd-41dd-b59e-cb38f2e8722d None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:51.862119 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c2dafec-b9bd-41dd-b59e-cb38f2e8722d None admin] PATH_INFO: `/v3/roles` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:51.862508 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c2dafec-b9bd-41dd-b59e-cb38f2e8722d None admin] RBAC: Adding query filter params (name=member) {{(pid=81437) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:53:51.862808 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c2dafec-b9bd-41dd-b59e-cb38f2e8722d None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:53:51.864209 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c2dafec-b9bd-41dd-b59e-cb38f2e8722d None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:53:51.866315 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:53:51.873485 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 220/440] 10.222.0.50 () {56 vars in 1101 bytes} [Tue Sep 13 15:53:51 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:53:51.879502 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-082fa29b-5dfe-41f9-84e5-3831c7873f00 None None] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:51.879785 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-082fa29b-5dfe-41f9-84e5-3831c7873f00 None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:51.880057 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-082fa29b-5dfe-41f9-84e5-3831c7873f00 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:51.884120 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87be40ab-a382-40e9-8a77-27f8b6dbd0be None None] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:53:51.897017 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.auth.core [None req-082fa29b-5dfe-41f9-84e5-3831c7873f00 None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81437) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:53:51.898179 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-082fa29b-5dfe-41f9-84e5-3831c7873f00 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:53:51.918068 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9c31841-5cda-4928-99fe-56bc0ce15c0c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:53:51.918653 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9c31841-5cda-4928-99fe-56bc0ce15c0c None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:51.918964 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9c31841-5cda-4928-99fe-56bc0ce15c0c None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:51.919244 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9c31841-5cda-4928-99fe-56bc0ce15c0c None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:51.963797 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 221/441] 10.222.0.50 () {58 vars in 1024 bytes} [Tue Sep 13 15:53:51 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 13 15:53:51.964697 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-082fa29b-5dfe-41f9-84e5-3831c7873f00 None None] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:53:51.999033 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9c31841-5cda-4928-99fe-56bc0ce15c0c None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:53:52.000558 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9c31841-5cda-4928-99fe-56bc0ce15c0c None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:53:52.064651 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:53:52.071723 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-082fa29b-5dfe-41f9-84e5-3831c7873f00 None None] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:53:52.102891 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:53:52.110920 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 221/442] 10.222.0.50 () {58 vars in 1282 bytes} [Tue Sep 13 15:53:51 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 334 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Sep 13 15:53:52.116449 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9c31841-5cda-4928-99fe-56bc0ce15c0c None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:53:52.116872 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca9d8d5c-149b-4f4b-96c9-05cb5bace01a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:53:52.117487 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca9d8d5c-149b-4f4b-96c9-05cb5bace01a None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:52.117771 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca9d8d5c-149b-4f4b-96c9-05cb5bace01a None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:52.118040 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca9d8d5c-149b-4f4b-96c9-05cb5bace01a None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:52.190868 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca9d8d5c-149b-4f4b-96c9-05cb5bace01a None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:53:52.192698 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca9d8d5c-149b-4f4b-96c9-05cb5bace01a None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:53:52.221433 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9c31841-5cda-4928-99fe-56bc0ce15c0c None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:53:52.256708 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ded3badf-1ee8-4123-8d2b-bfe0674d5a04 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:53:52.257337 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ded3badf-1ee8-4123-8d2b-bfe0674d5a04 None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:52.257618 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ded3badf-1ee8-4123-8d2b-bfe0674d5a04 None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:52.257893 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ded3badf-1ee8-4123-8d2b-bfe0674d5a04 None admin] PATH_INFO: `/v3/projects/9ec1adca55204df2b653531664b60123` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:52.259042 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ded3badf-1ee8-4123-8d2b-bfe0674d5a04 None admin] RBAC: Authorizing `identity:get_project(project_id=9ec1adca55204df2b653531664b60123)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:53:52.260821 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ded3badf-1ee8-4123-8d2b-bfe0674d5a04 None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:53:52.262433 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 222/443] 10.222.0.50 () {56 vars in 1186 bytes} [Tue Sep 13 15:53:52 2022] GET /identity/v3/projects/9ec1adca55204df2b653531664b60123 => generated 302 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:53:52.263220 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ded3badf-1ee8-4123-8d2b-bfe0674d5a04 None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:53:52.290809 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:53:52.298659 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 222/444] 10.222.0.50 () {58 vars in 1282 bytes} [Tue Sep 13 15:53:51 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 334 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 13 15:53:52.299604 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca9d8d5c-149b-4f4b-96c9-05cb5bace01a None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:53:52.326824 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:53:52.367760 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ded3badf-1ee8-4123-8d2b-bfe0674d5a04 None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:53:52.398482 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca9d8d5c-149b-4f4b-96c9-05cb5bace01a None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:53:52.401961 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27e6bd07-8ad0-4bb2-bfa4-faa1c354dbef None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:53:52.401961 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27e6bd07-8ad0-4bb2-bfa4-faa1c354dbef None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:52.402197 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27e6bd07-8ad0-4bb2-bfa4-faa1c354dbef None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:52.402404 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27e6bd07-8ad0-4bb2-bfa4-faa1c354dbef None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:52.412273 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27e6bd07-8ad0-4bb2-bfa4-faa1c354dbef None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 13 15:53:52.417088 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application [None req-27e6bd07-8ad0-4bb2-bfa4-faa1c354dbef None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 13 15:53:52.417088 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:53:52.417088 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:53:52.417088 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:53:52.417088 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:53:52.417088 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:53:52.417088 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:53:52.417088 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:53:52.417088 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:53:52.417088 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:53:52.417088 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:53:52.417088 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:53:52.417088 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 13 15:53:52.417088 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 13 15:53:52.417088 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 13 15:53:52.417088 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 13 15:53:52.418041 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 13 15:53:52.418041 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 13 15:53:52.418041 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 13 15:53:52.418041 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 13 15:53:52.418041 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:53:52.418041 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:53:52.418041 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:53:52.418041 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:53:52.418041 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:53:52.418041 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:53:52.418041 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:53:52.418041 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application with Lock( Sep 13 15:53:52.418041 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:53:52.418041 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:53:52.418041 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:53:52.418041 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:53:52.418850 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:53:52.418850 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:53:52.418850 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:53:52.418850 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:53:52.418850 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 13 15:53:52.418850 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 13 15:53:52.418850 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 13 15:53:52.418850 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 13 15:53:52.418850 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 13 15:53:52.418850 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 13 15:53:52.418850 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 13 15:53:52.418850 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Sep 13 15:53:52.418850 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 223/445] 10.222.0.50 () {56 vars in 1096 bytes} [Tue Sep 13 15:53:52 2022] GET /identity/v3/roles/admin => generated 83 bytes in 155 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:53:52.419547 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27e6bd07-8ad0-4bb2-bfa4-faa1c354dbef None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:53:52.433710 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b5b351d-9e7b-489a-b24e-6161ea3c43d0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:53:52.434341 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b5b351d-9e7b-489a-b24e-6161ea3c43d0 None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:52.434613 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b5b351d-9e7b-489a-b24e-6161ea3c43d0 None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:52.434886 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b5b351d-9e7b-489a-b24e-6161ea3c43d0 None admin] PATH_INFO: `/v3/groups/acdd1f9701134c2e9b5d957895b30027` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:52.436285 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b5b351d-9e7b-489a-b24e-6161ea3c43d0 None admin] RBAC: Authorizing `identity:get_group(group_id=acdd1f9701134c2e9b5d957895b30027)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:53:52.437718 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b5b351d-9e7b-489a-b24e-6161ea3c43d0 None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:53:52.439287 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 223/446] 10.222.0.50 () {56 vars in 1180 bytes} [Tue Sep 13 15:53:52 2022] GET /identity/v3/groups/acdd1f9701134c2e9b5d957895b30027 => generated 226 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:53:52.440032 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b5b351d-9e7b-489a-b24e-6161ea3c43d0 None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:53:52.512378 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:53:52.524723 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27e6bd07-8ad0-4bb2-bfa4-faa1c354dbef None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:53:52.533879 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:53:52.539872 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b5b351d-9e7b-489a-b24e-6161ea3c43d0 None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:53:52.559538 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5d41cc3-e685-4b4a-96be-b25c3d42e808 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:53:52.560171 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5d41cc3-e685-4b4a-96be-b25c3d42e808 None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:52.560449 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5d41cc3-e685-4b4a-96be-b25c3d42e808 None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:52.560718 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5d41cc3-e685-4b4a-96be-b25c3d42e808 None admin] PATH_INFO: `/v3/projects/9ec1adca55204df2b653531664b60123` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:52.561831 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5d41cc3-e685-4b4a-96be-b25c3d42e808 None admin] RBAC: Authorizing `identity:get_project(project_id=9ec1adca55204df2b653531664b60123)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:53:52.563289 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5d41cc3-e685-4b4a-96be-b25c3d42e808 None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:53:52.564889 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 224/447] 10.222.0.50 () {56 vars in 1186 bytes} [Tue Sep 13 15:53:52 2022] GET /identity/v3/projects/9ec1adca55204df2b653531664b60123 => generated 302 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:53:52.565660 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5d41cc3-e685-4b4a-96be-b25c3d42e808 None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:53:52.573245 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-622a8583-5d9e-4bcd-99bd-7b96e4736443 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:53:52.573852 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-622a8583-5d9e-4bcd-99bd-7b96e4736443 None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:52.574133 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-622a8583-5d9e-4bcd-99bd-7b96e4736443 None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:52.574407 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-622a8583-5d9e-4bcd-99bd-7b96e4736443 None admin] PATH_INFO: `/v3/roles` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:52.575098 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-622a8583-5d9e-4bcd-99bd-7b96e4736443 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81437) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:53:52.575400 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-622a8583-5d9e-4bcd-99bd-7b96e4736443 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:53:52.576765 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-622a8583-5d9e-4bcd-99bd-7b96e4736443 None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:53:52.584764 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 224/448] 10.222.0.50 () {56 vars in 1099 bytes} [Tue Sep 13 15:53:52 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 145 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:53:52.585539 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-622a8583-5d9e-4bcd-99bd-7b96e4736443 None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:53:52.663262 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5d41cc3-e685-4b4a-96be-b25c3d42e808 None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:53:52.686514 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-622a8583-5d9e-4bcd-99bd-7b96e4736443 None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:53:52.697237 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0331b7f7-bb0b-4062-8e26-d99090ea0ba3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:53:52.698578 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0331b7f7-bb0b-4062-8e26-d99090ea0ba3 None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:52.698854 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0331b7f7-bb0b-4062-8e26-d99090ea0ba3 None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:52.699149 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0331b7f7-bb0b-4062-8e26-d99090ea0ba3 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:52.699865 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0331b7f7-bb0b-4062-8e26-d99090ea0ba3 None admin] RBAC: Adding query filter params (group.id=acdd1f9701134c2e9b5d957895b30027, role.id=f74e3f6c20bb426ca030f211c3a9aef5, scope.project.id=9ec1adca55204df2b653531664b60123) {{(pid=81438) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:53:52.700176 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0331b7f7-bb0b-4062-8e26-d99090ea0ba3 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:53:52.701603 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0331b7f7-bb0b-4062-8e26-d99090ea0ba3 None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:53:52.707387 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 225/449] 10.222.0.50 () {56 vars in 1376 bytes} [Tue Sep 13 15:53:52 2022] GET /identity/v3/role_assignments?group.id=acdd1f9701134c2e9b5d957895b30027&scope.project.id=9ec1adca55204df2b653531664b60123&role.id=f74e3f6c20bb426ca030f211c3a9aef5 => generated 611 bytes in 142 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:53:52.708209 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0331b7f7-bb0b-4062-8e26-d99090ea0ba3 None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:53:52.724078 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a1aa8cb-4b4e-4ae0-b11c-d5958f20b588 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:53:52.724725 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a1aa8cb-4b4e-4ae0-b11c-d5958f20b588 None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:52.725003 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a1aa8cb-4b4e-4ae0-b11c-d5958f20b588 None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:52.725387 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a1aa8cb-4b4e-4ae0-b11c-d5958f20b588 None admin] PATH_INFO: `/v3/groups/acdd1f9701134c2e9b5d957895b30027` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:52.726665 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a1aa8cb-4b4e-4ae0-b11c-d5958f20b588 None admin] RBAC: Authorizing `identity:get_group(group_id=acdd1f9701134c2e9b5d957895b30027)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:53:52.728213 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a1aa8cb-4b4e-4ae0-b11c-d5958f20b588 None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:53:52.735430 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 225/450] 10.222.0.50 () {56 vars in 1180 bytes} [Tue Sep 13 15:53:52 2022] GET /identity/v3/groups/acdd1f9701134c2e9b5d957895b30027 => generated 226 bytes in 145 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:53:52.735673 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:53:52.737297 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a1aa8cb-4b4e-4ae0-b11c-d5958f20b588 None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:53:52.769130 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:53:52.828958 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0331b7f7-bb0b-4062-8e26-d99090ea0ba3 None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:53:52.853391 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a1aa8cb-4b4e-4ae0-b11c-d5958f20b588 None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:53:52.866725 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-891c7abe-0ffd-4895-867f-587524925e54 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:53:52.867398 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-891c7abe-0ffd-4895-867f-587524925e54 None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:52.867675 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-891c7abe-0ffd-4895-867f-587524925e54 None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:52.867945 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-891c7abe-0ffd-4895-867f-587524925e54 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:52.894549 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b87c9d57-62a0-4034-b299-48576b01d9a6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:53:52.895790 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b87c9d57-62a0-4034-b299-48576b01d9a6 None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:52.896229 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b87c9d57-62a0-4034-b299-48576b01d9a6 None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:52.896507 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b87c9d57-62a0-4034-b299-48576b01d9a6 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:52.897241 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b87c9d57-62a0-4034-b299-48576b01d9a6 None admin] RBAC: Adding query filter params (group.id=acdd1f9701134c2e9b5d957895b30027, role.id=15db4aaca3264ae29bde38903ba43471, scope.project.id=9ec1adca55204df2b653531664b60123) {{(pid=81437) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:53:52.897554 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b87c9d57-62a0-4034-b299-48576b01d9a6 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:53:52.899037 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b87c9d57-62a0-4034-b299-48576b01d9a6 None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:53:52.905322 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 226/451] 10.222.0.50 () {56 vars in 1376 bytes} [Tue Sep 13 15:53:52 2022] GET /identity/v3/role_assignments?group.id=acdd1f9701134c2e9b5d957895b30027&scope.project.id=9ec1adca55204df2b653531664b60123&role.id=15db4aaca3264ae29bde38903ba43471 => generated 611 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 13 15:53:52.947044 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-891c7abe-0ffd-4895-867f-587524925e54 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:53:52.949481 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-891c7abe-0ffd-4895-867f-587524925e54 None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:53:52.972744 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:53:53.055340 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 226/452] 10.222.0.50 () {58 vars in 1282 bytes} [Tue Sep 13 15:53:52 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 347 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 13 15:53:53.062267 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b87c9d57-62a0-4034-b299-48576b01d9a6 None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:53:53.155620 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:53:53.165443 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b87c9d57-62a0-4034-b299-48576b01d9a6 None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:53:53.201046 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b38a9244-0678-4e36-9eef-46ae486717b4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:53:53.202011 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b38a9244-0678-4e36-9eef-46ae486717b4 None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:53.202295 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b38a9244-0678-4e36-9eef-46ae486717b4 None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:53.202571 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b38a9244-0678-4e36-9eef-46ae486717b4 None admin] PATH_INFO: `/v3/projects/555bee08a3fa4d08bdd8ecc881adc45b` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:53.203766 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b38a9244-0678-4e36-9eef-46ae486717b4 None admin] RBAC: Authorizing `identity:get_project(project_id=555bee08a3fa4d08bdd8ecc881adc45b)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:53:53.205211 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b38a9244-0678-4e36-9eef-46ae486717b4 None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:53:53.206837 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 227/453] 10.222.0.50 () {56 vars in 1186 bytes} [Tue Sep 13 15:53:53 2022] GET /identity/v3/projects/555bee08a3fa4d08bdd8ecc881adc45b => generated 344 bytes in 145 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:53:53.212464 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-891c7abe-0ffd-4895-867f-587524925e54 None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:53:53.321247 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-891c7abe-0ffd-4895-867f-587524925e54 None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:53:53.341401 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:53:53.356532 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f36c6074-fbe8-466f-9b1c-5003e2fa03b1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:53:53.357170 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f36c6074-fbe8-466f-9b1c-5003e2fa03b1 None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:53.357449 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f36c6074-fbe8-466f-9b1c-5003e2fa03b1 None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:53.357723 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f36c6074-fbe8-466f-9b1c-5003e2fa03b1 None admin] PATH_INFO: `/v3/groups/58a500e5c4724a8eb4916f4251241715` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:53.358831 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f36c6074-fbe8-466f-9b1c-5003e2fa03b1 None admin] RBAC: Authorizing `identity:get_group(group_id=58a500e5c4724a8eb4916f4251241715)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:53:53.360305 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f36c6074-fbe8-466f-9b1c-5003e2fa03b1 None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:53:53.361867 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 227/454] 10.222.0.50 () {56 vars in 1180 bytes} [Tue Sep 13 15:53:53 2022] GET /identity/v3/groups/58a500e5c4724a8eb4916f4251241715 => generated 229 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:53:53.367796 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b38a9244-0678-4e36-9eef-46ae486717b4 None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:53:53.468183 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b38a9244-0678-4e36-9eef-46ae486717b4 None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:53:53.501089 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3784d8fd-bfa0-492b-b77a-f3ade5013fba None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:53:53.501690 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3784d8fd-bfa0-492b-b77a-f3ade5013fba None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:53.501962 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3784d8fd-bfa0-492b-b77a-f3ade5013fba None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:53.502231 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3784d8fd-bfa0-492b-b77a-f3ade5013fba None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:53.502968 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3784d8fd-bfa0-492b-b77a-f3ade5013fba None admin] RBAC: Adding query filter params (group.id=58a500e5c4724a8eb4916f4251241715, role.id=031d4e1a23e34f71858fc4a6dfbd4ccb, scope.project.id=555bee08a3fa4d08bdd8ecc881adc45b) {{(pid=81437) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:53:53.503275 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3784d8fd-bfa0-492b-b77a-f3ade5013fba None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:53:53.504728 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3784d8fd-bfa0-492b-b77a-f3ade5013fba None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:53:53.511301 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 228/455] 10.222.0.50 () {56 vars in 1376 bytes} [Tue Sep 13 15:53:53 2022] GET /identity/v3/role_assignments?group.id=58a500e5c4724a8eb4916f4251241715&scope.project.id=555bee08a3fa4d08bdd8ecc881adc45b&role.id=031d4e1a23e34f71858fc4a6dfbd4ccb => generated 611 bytes in 144 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:53:56.164599 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b0228f1-bcf0-4592-bab4-3735c8c29782 None None] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:56.165137 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b0228f1-bcf0-4592-bab4-3735c8c29782 None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:56.165137 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b0228f1-bcf0-4592-bab4-3735c8c29782 None None] PATH_INFO: `/` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:56.167278 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 228/456] 10.222.0.50 () {54 vars in 927 bytes} [Tue Sep 13 15:53:56 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:53:56.172593 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3de631cc-4f6c-4d93-a8ef-a78a4bec28c9 None None] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:56.173068 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3de631cc-4f6c-4d93-a8ef-a78a4bec28c9 None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:56.173168 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3de631cc-4f6c-4d93-a8ef-a78a4bec28c9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:56.198977 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.auth.core [None req-3de631cc-4f6c-4d93-a8ef-a78a4bec28c9 None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81437) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:53:56.200314 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-3de631cc-4f6c-4d93-a8ef-a78a4bec28c9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:53:56.221016 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:53:56.289362 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 229/457] 10.222.0.50 () {58 vars in 1024 bytes} [Tue Sep 13 15:53:56 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 118 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 13 15:53:56.480464 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdd4428c-f808-4b46-ab67-381f0b094b20 None None] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:56.481123 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdd4428c-f808-4b46-ab67-381f0b094b20 None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:56.481606 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdd4428c-f808-4b46-ab67-381f0b094b20 None None] PATH_INFO: `/` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:56.483456 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 229/458] 10.222.0.50 () {54 vars in 927 bytes} [Tue Sep 13 15:53:56 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:53:56.488499 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2201ff4-7ac0-44ad-9a77-c56e4360d15b None None] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:56.488781 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2201ff4-7ac0-44ad-9a77-c56e4360d15b None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:56.489052 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2201ff4-7ac0-44ad-9a77-c56e4360d15b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:56.528074 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.auth.core [None req-e2201ff4-7ac0-44ad-9a77-c56e4360d15b None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81437) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:53:56.529300 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-e2201ff4-7ac0-44ad-9a77-c56e4360d15b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:53:56.619587 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 230/459] 10.222.0.50 () {58 vars in 1024 bytes} [Tue Sep 13 15:53:56 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 132 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 13 15:53:56.644827 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdd4428c-f808-4b46-ab67-381f0b094b20 None None] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:53:56.656233 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b8e2652-faa4-4d16-bc1d-86ce59830aa1 None None] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:56.656507 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b8e2652-faa4-4d16-bc1d-86ce59830aa1 None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:56.656788 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b8e2652-faa4-4d16-bc1d-86ce59830aa1 None None] PATH_INFO: `/` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:56.658427 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 231/460] 10.222.0.50 () {54 vars in 927 bytes} [Tue Sep 13 15:53:56 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:53:56.663344 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0188ae6d-54d7-4f53-85c9-683d8b1e428b None None] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:56.663638 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0188ae6d-54d7-4f53-85c9-683d8b1e428b None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:56.663891 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0188ae6d-54d7-4f53-85c9-683d8b1e428b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:56.683227 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.auth.core [None req-0188ae6d-54d7-4f53-85c9-683d8b1e428b None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81437) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:53:56.685605 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-0188ae6d-54d7-4f53-85c9-683d8b1e428b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:53:56.809291 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 232/461] 10.222.0.50 () {58 vars in 1024 bytes} [Tue Sep 13 15:53:56 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 146 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 13 15:53:56.827452 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdd4428c-f808-4b46-ab67-381f0b094b20 None None] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:53:56.883036 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7fce935-4752-4d70-b245-8049022761cb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:53:56.883668 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7fce935-4752-4d70-b245-8049022761cb None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:56.883939 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7fce935-4752-4d70-b245-8049022761cb None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:56.884207 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7fce935-4752-4d70-b245-8049022761cb None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:56.903095 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7fce935-4752-4d70-b245-8049022761cb None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:53:56.904812 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7fce935-4752-4d70-b245-8049022761cb None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:53:56.919450 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application [None req-f7fce935-4752-4d70-b245-8049022761cb None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:53:56.919450 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:53:56.919450 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:53:56.919450 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:53:56.919450 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:53:56.919450 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:53:56.919450 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:53:56.919450 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:53:56.919450 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:53:56.919450 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:53:56.919450 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:53:56.919450 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:53:56.919450 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 13 15:53:56.919450 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 13 15:53:56.919450 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 13 15:53:56.919450 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 13 15:53:56.920387 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:53:56.920387 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:53:56.920387 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:53:56.920387 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:53:56.920387 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:53:56.920387 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:53:56.920387 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:53:56.920387 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application with Lock( Sep 13 15:53:56.920387 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:53:56.920387 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:53:56.920387 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:53:56.920387 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:53:56.920387 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:53:56.920387 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:53:56.920387 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:53:56.920387 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:53:56.920387 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 13 15:53:56.921255 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 13 15:53:56.921255 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:53:56.921255 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Sep 13 15:53:56.921255 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 230/462] 10.222.0.50 () {56 vars in 1108 bytes} [Tue Sep 13 15:53:56 2022] GET /identity/v3/domains/Default => generated 87 bytes in 276 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:53:56.928213 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0188ae6d-54d7-4f53-85c9-683d8b1e428b None None] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:53:56.952030 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7fce935-4752-4d70-b245-8049022761cb None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:53:56.952479 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:53:57.026659 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:53:57.108691 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0188ae6d-54d7-4f53-85c9-683d8b1e428b None None] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:53:57.142148 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7fce935-4752-4d70-b245-8049022761cb None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:53:57.161496 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cf1caac-4446-4b04-a598-8f76f4482495 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:53:57.162472 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cf1caac-4446-4b04-a598-8f76f4482495 None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:57.162889 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cf1caac-4446-4b04-a598-8f76f4482495 None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:57.163430 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cf1caac-4446-4b04-a598-8f76f4482495 None admin] PATH_INFO: `/v3/domains` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:57.164216 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4cf1caac-4446-4b04-a598-8f76f4482495 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81437) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:53:57.164654 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4cf1caac-4446-4b04-a598-8f76f4482495 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:53:57.166378 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4cf1caac-4446-4b04-a598-8f76f4482495 None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:53:57.181531 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 233/463] 10.222.0.50 () {56 vars in 1109 bytes} [Tue Sep 13 15:53:56 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 254 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:53:57.187107 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cf1caac-4446-4b04-a598-8f76f4482495 None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:53:57.187740 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27394e6a-eeab-47b4-8b19-8c26d81e9c55 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:53:57.188388 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27394e6a-eeab-47b4-8b19-8c26d81e9c55 None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:57.188663 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27394e6a-eeab-47b4-8b19-8c26d81e9c55 None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:57.188933 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27394e6a-eeab-47b4-8b19-8c26d81e9c55 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:57.201959 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27394e6a-eeab-47b4-8b19-8c26d81e9c55 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:53:57.203458 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27394e6a-eeab-47b4-8b19-8c26d81e9c55 None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:53:57.220054 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application [None req-27394e6a-eeab-47b4-8b19-8c26d81e9c55 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:53:57.220054 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:53:57.220054 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:53:57.220054 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:53:57.220054 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:53:57.220054 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:53:57.220054 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:53:57.220054 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:53:57.220054 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:53:57.220054 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:53:57.220054 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:53:57.220054 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:53:57.220054 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 13 15:53:57.220054 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 13 15:53:57.220054 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 13 15:53:57.220054 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 13 15:53:57.221042 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:53:57.221042 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:53:57.221042 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:53:57.221042 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:53:57.221042 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:53:57.221042 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:53:57.221042 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:53:57.221042 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application with Lock( Sep 13 15:53:57.221042 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:53:57.221042 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:53:57.221042 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:53:57.221042 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:53:57.221042 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:53:57.221042 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:53:57.221042 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:53:57.221042 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:53:57.221042 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 13 15:53:57.221866 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 13 15:53:57.221866 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:53:57.221866 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Sep 13 15:53:57.222032 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 231/464] 10.222.0.50 () {56 vars in 1108 bytes} [Tue Sep 13 15:53:56 2022] GET /identity/v3/domains/Default => generated 87 bytes in 270 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 13 15:53:57.222821 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27394e6a-eeab-47b4-8b19-8c26d81e9c55 None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:53:57.280876 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:53:57.313256 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cf1caac-4446-4b04-a598-8f76f4482495 None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:53:57.327331 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:53:57.336638 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27394e6a-eeab-47b4-8b19-8c26d81e9c55 None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:53:57.355246 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-429b602d-2957-435c-85d9-fc866baf379f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:53:57.355889 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-429b602d-2957-435c-85d9-fc866baf379f None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:57.356166 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-429b602d-2957-435c-85d9-fc866baf379f None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:57.356438 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-429b602d-2957-435c-85d9-fc866baf379f None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:57.371552 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd3d8bb6-0ace-4efb-bf75-ba0dbdb79946 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:53:57.371712 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-429b602d-2957-435c-85d9-fc866baf379f None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:53:57.372159 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd3d8bb6-0ace-4efb-bf75-ba0dbdb79946 None admin] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:57.372441 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd3d8bb6-0ace-4efb-bf75-ba0dbdb79946 None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:57.372715 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd3d8bb6-0ace-4efb-bf75-ba0dbdb79946 None admin] PATH_INFO: `/v3/users` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:57.373006 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-429b602d-2957-435c-85d9-fc866baf379f None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:53:57.373393 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd3d8bb6-0ace-4efb-bf75-ba0dbdb79946 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:53:57.374767 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd3d8bb6-0ace-4efb-bf75-ba0dbdb79946 None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:53:57.390657 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application [None req-429b602d-2957-435c-85d9-fc866baf379f None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:53:57.390657 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:53:57.390657 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:53:57.390657 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:53:57.390657 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:53:57.390657 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:53:57.390657 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:53:57.390657 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:53:57.390657 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:53:57.390657 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:53:57.390657 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:53:57.390657 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:53:57.390657 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 13 15:53:57.390657 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 13 15:53:57.390657 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 13 15:53:57.390657 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 13 15:53:57.391653 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:53:57.391653 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:53:57.391653 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:53:57.391653 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:53:57.391653 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:53:57.391653 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:53:57.391653 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:53:57.391653 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application with Lock( Sep 13 15:53:57.391653 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:53:57.391653 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:53:57.391653 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:53:57.391653 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:53:57.391653 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:53:57.391653 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:53:57.391653 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:53:57.391653 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:53:57.391653 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 13 15:53:57.392497 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 13 15:53:57.392497 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:53:57.392497 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application Sep 13 15:53:57.394472 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 234/465] 10.222.0.50 () {56 vars in 1108 bytes} [Tue Sep 13 15:53:57 2022] GET /identity/v3/domains/Default => generated 87 bytes in 212 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:53:57.395835 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-095750dc-031e-42f1-9f33-76cc664fea68 None None] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:57.396122 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-095750dc-031e-42f1-9f33-76cc664fea68 None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:57.396400 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-095750dc-031e-42f1-9f33-76cc664fea68 None None] PATH_INFO: `/` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:57.402469 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 235/466] 10.222.0.50 () {54 vars in 927 bytes} [Tue Sep 13 15:53:57 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:53:57.402888 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 232/467] 10.222.0.50 () {60 vars in 1132 bytes} [Tue Sep 13 15:53:57 2022] POST /identity/v3/users => generated 246 bytes in 180 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 15:53:57.403337 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-095750dc-031e-42f1-9f33-76cc664fea68 None None] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:53:57.404148 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd3d8bb6-0ace-4efb-bf75-ba0dbdb79946 None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:53:57.536897 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-095750dc-031e-42f1-9f33-76cc664fea68 None None] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:53:57.545757 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd3d8bb6-0ace-4efb-bf75-ba0dbdb79946 None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:53:57.570120 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:53:57.584923 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01dbe7ef-82ce-47a4-ab7e-d44f6961ea12 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:53:57.585830 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01dbe7ef-82ce-47a4-ab7e-d44f6961ea12 None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:57.586300 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01dbe7ef-82ce-47a4-ab7e-d44f6961ea12 None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:57.586751 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01dbe7ef-82ce-47a4-ab7e-d44f6961ea12 None admin] PATH_INFO: `/v3/domains` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:57.587512 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e414343c-fc2c-4706-b348-64f97854246e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:53:57.587820 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01dbe7ef-82ce-47a4-ab7e-d44f6961ea12 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81437) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:53:57.588190 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e414343c-fc2c-4706-b348-64f97854246e None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:57.588473 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e414343c-fc2c-4706-b348-64f97854246e None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:57.588743 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e414343c-fc2c-4706-b348-64f97854246e None admin] PATH_INFO: `/v3/domains` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:57.588987 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01dbe7ef-82ce-47a4-ab7e-d44f6961ea12 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:53:57.589390 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e414343c-fc2c-4706-b348-64f97854246e None admin] RBAC: Adding query filter params (name=Default) {{(pid=81438) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:53:57.589696 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e414343c-fc2c-4706-b348-64f97854246e None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:53:57.590397 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01dbe7ef-82ce-47a4-ab7e-d44f6961ea12 None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:53:57.591123 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e414343c-fc2c-4706-b348-64f97854246e None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:53:57.604222 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 233/468] 10.222.0.50 () {56 vars in 1109 bytes} [Tue Sep 13 15:53:57 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 200 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:53:57.604554 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 236/469] 10.222.0.50 () {56 vars in 1109 bytes} [Tue Sep 13 15:53:57 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 201 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:53:57.605363 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8d40443-0281-4df8-8a81-1223d5bf5695 None None] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:57.605600 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb6aaf52-bd2e-451e-bb4b-7e35b7027f62 None None] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:57.605886 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb6aaf52-bd2e-451e-bb4b-7e35b7027f62 None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:57.606160 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb6aaf52-bd2e-451e-bb4b-7e35b7027f62 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:57.609949 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8d40443-0281-4df8-8a81-1223d5bf5695 None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:57.610237 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8d40443-0281-4df8-8a81-1223d5bf5695 None None] PATH_INFO: `/` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:57.611583 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 237/470] 10.222.0.50 () {54 vars in 927 bytes} [Tue Sep 13 15:53:57 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:53:57.612305 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8d40443-0281-4df8-8a81-1223d5bf5695 None None] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:53:57.631991 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.auth.core [None req-cb6aaf52-bd2e-451e-bb4b-7e35b7027f62 None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81438) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:53:57.633274 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-cb6aaf52-bd2e-451e-bb4b-7e35b7027f62 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:53:57.636354 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:53:57.708676 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 234/471] 10.222.0.50 () {58 vars in 1024 bytes} [Tue Sep 13 15:53:57 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 13 15:53:57.709500 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb6aaf52-bd2e-451e-bb4b-7e35b7027f62 None None] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:53:57.737119 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8d40443-0281-4df8-8a81-1223d5bf5695 None None] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:53:57.775500 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23b6bdb8-b641-4bd1-a875-3aa31f8ea56b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:53:57.776576 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23b6bdb8-b641-4bd1-a875-3aa31f8ea56b None admin] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:57.777195 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23b6bdb8-b641-4bd1-a875-3aa31f8ea56b None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:57.777652 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23b6bdb8-b641-4bd1-a875-3aa31f8ea56b None admin] PATH_INFO: `/v3/users` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:57.778503 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23b6bdb8-b641-4bd1-a875-3aa31f8ea56b None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:53:57.780220 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23b6bdb8-b641-4bd1-a875-3aa31f8ea56b None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:53:57.809718 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 238/472] 10.222.0.50 () {60 vars in 1132 bytes} [Tue Sep 13 15:53:57 2022] POST /identity/v3/users => generated 248 bytes in 197 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 15:53:57.811139 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-555ba033-7e0d-4554-baea-bd5843ecf382 None None] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:57.811522 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-555ba033-7e0d-4554-baea-bd5843ecf382 None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:57.811990 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-555ba033-7e0d-4554-baea-bd5843ecf382 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:57.823945 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb6aaf52-bd2e-451e-bb4b-7e35b7027f62 None None] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:53:57.832918 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.auth.core [None req-555ba033-7e0d-4554-baea-bd5843ecf382 None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81437) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:53:57.834013 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-555ba033-7e0d-4554-baea-bd5843ecf382 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:53:57.865371 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b17a104c-5dc2-4405-b4e9-85eabf3e6523 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:53:57.866336 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b17a104c-5dc2-4405-b4e9-85eabf3e6523 None admin] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:57.866821 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b17a104c-5dc2-4405-b4e9-85eabf3e6523 None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:57.867325 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b17a104c-5dc2-4405-b4e9-85eabf3e6523 None admin] PATH_INFO: `/v3/users` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:57.868184 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b17a104c-5dc2-4405-b4e9-85eabf3e6523 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:53:57.870477 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b17a104c-5dc2-4405-b4e9-85eabf3e6523 None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:53:57.881899 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:53:57.901226 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 235/473] 10.222.0.50 () {60 vars in 1132 bytes} [Tue Sep 13 15:53:57 2022] POST /identity/v3/users => generated 248 bytes in 192 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 15:53:57.910029 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:53:57.966660 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 239/474] 10.222.0.50 () {58 vars in 1024 bytes} [Tue Sep 13 15:53:57 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 156 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 13 15:53:58.115746 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b17a104c-5dc2-4405-b4e9-85eabf3e6523 None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:53:58.228415 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b17a104c-5dc2-4405-b4e9-85eabf3e6523 None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:53:58.241235 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-555ba033-7e0d-4554-baea-bd5843ecf382 None None] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:53:58.268994 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d7bae79-29c9-48bc-b226-1b0676440b58 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:53:58.269889 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d7bae79-29c9-48bc-b226-1b0676440b58 None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:58.270343 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d7bae79-29c9-48bc-b226-1b0676440b58 None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:58.271070 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d7bae79-29c9-48bc-b226-1b0676440b58 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:58.285798 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d7bae79-29c9-48bc-b226-1b0676440b58 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:53:58.287720 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d7bae79-29c9-48bc-b226-1b0676440b58 None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:53:58.307912 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application [None req-7d7bae79-29c9-48bc-b226-1b0676440b58 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:53:58.307912 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:53:58.307912 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:53:58.307912 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:53:58.307912 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:53:58.307912 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:53:58.307912 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:53:58.307912 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:53:58.307912 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:53:58.307912 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:53:58.307912 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:53:58.307912 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:53:58.307912 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 13 15:53:58.307912 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 13 15:53:58.307912 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 13 15:53:58.307912 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 13 15:53:58.309012 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:53:58.309012 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:53:58.309012 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:53:58.309012 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:53:58.309012 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:53:58.309012 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:53:58.309012 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:53:58.309012 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application with Lock( Sep 13 15:53:58.309012 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:53:58.309012 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:53:58.309012 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:53:58.309012 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:53:58.309012 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:53:58.309012 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:53:58.309012 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:53:58.309012 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:53:58.309012 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 13 15:53:58.310071 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 13 15:53:58.310071 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:53:58.310071 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Sep 13 15:53:58.311841 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 236/475] 10.222.0.50 () {56 vars in 1108 bytes} [Tue Sep 13 15:53:58 2022] GET /identity/v3/domains/Default => generated 87 bytes in 197 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:53:58.319469 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d7bae79-29c9-48bc-b226-1b0676440b58 None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:53:58.376336 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:INSERT=1,keystone:SELECT=99 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:53:58.377403 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-555ba033-7e0d-4554-baea-bd5843ecf382 None None] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:53:58.451181 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-085d20c3-0a9a-4fe0-a325-83f9eefff70d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:53:58.452221 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-085d20c3-0a9a-4fe0-a325-83f9eefff70d None admin] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:58.452701 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-085d20c3-0a9a-4fe0-a325-83f9eefff70d None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:58.453189 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-085d20c3-0a9a-4fe0-a325-83f9eefff70d None admin] PATH_INFO: `/v3/roles` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:58.454098 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-085d20c3-0a9a-4fe0-a325-83f9eefff70d None admin] RBAC: Authorizing `identity:create_role()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:53:58.455708 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-085d20c3-0a9a-4fe0-a325-83f9eefff70d None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:53:58.472172 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.sql.core [None req-085d20c3-0a9a-4fe0-a325-83f9eefff70d None admin] Conflict role: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'anotherrole-<>' for key 'role.ixu_role_name_domain_id'") Sep 13 15:53:58.472172 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [SQL: INSERT INTO `role` (id, name, description, extra) VALUES (%(id)s, %(name)s, %(description)s, %(extra)s)] Sep 13 15:53:58.472172 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [parameters: {'id': '48de1c01d52a4586b0f45e0008983108', 'name': 'anotherrole', 'description': None, 'extra': '{}'}] Sep 13 15:53:58.472172 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: (Background on this error at: https://sqlalche.me/e/14/gkpj) {{(pid=81437) wrapper /opt/stack/keystone/keystone/common/sql/core.py:533}} Sep 13 15:53:58.490604 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application [None req-085d20c3-0a9a-4fe0-a325-83f9eefff70d None admin] Conflict occurred attempting to store role - Duplicate entry found with name anotherrole.: keystone.exception.Conflict: Conflict occurred attempting to store role - Duplicate entry found with name anotherrole. Sep 13 15:53:58.490604 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:53:58.490604 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Sep 13 15:53:58.490604 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application self.dialect.do_execute( Sep 13 15:53:58.490604 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Sep 13 15:53:58.490604 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Sep 13 15:53:58.490604 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Sep 13 15:53:58.490604 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application result = self._query(query) Sep 13 15:53:58.490604 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Sep 13 15:53:58.490604 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application conn.query(q) Sep 13 15:53:58.490604 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Sep 13 15:53:58.490604 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Sep 13 15:53:58.490604 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Sep 13 15:53:58.490604 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application result.read() Sep 13 15:53:58.490604 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Sep 13 15:53:58.491540 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Sep 13 15:53:58.491540 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Sep 13 15:53:58.491540 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application packet.raise_for_error() Sep 13 15:53:58.491540 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Sep 13 15:53:58.491540 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Sep 13 15:53:58.491540 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Sep 13 15:53:58.491540 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Sep 13 15:53:58.491540 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application pymysql.err.IntegrityError: (1062, "Duplicate entry 'anotherrole-<>' for key 'role.ixu_role_name_domain_id'") Sep 13 15:53:58.491540 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application Sep 13 15:53:58.491540 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application The above exception was the direct cause of the following exception: Sep 13 15:53:58.491540 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application Sep 13 15:53:58.491540 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:53:58.491540 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 528, in wrapper Sep 13 15:53:58.491540 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return method(*args, **kwargs) Sep 13 15:53:58.491540 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 33, in create_role Sep 13 15:53:58.491540 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return ref.to_dict() Sep 13 15:53:58.491540 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Sep 13 15:53:58.491540 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application next(self.gen) Sep 13 15:53:58.492451 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 1060, in _transaction_scope Sep 13 15:53:58.492451 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application yield resource Sep 13 15:53:58.492451 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Sep 13 15:53:58.492451 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application next(self.gen) Sep 13 15:53:58.492451 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 662, in _session Sep 13 15:53:58.492451 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application self.session.rollback() Sep 13 15:53:58.492451 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Sep 13 15:53:58.492451 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application self.force_reraise() Sep 13 15:53:58.492451 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Sep 13 15:53:58.492451 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application raise self.value Sep 13 15:53:58.492451 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 659, in _session Sep 13 15:53:58.492451 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application self._end_session_transaction(self.session) Sep 13 15:53:58.492451 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 687, in _end_session_transaction Sep 13 15:53:58.492451 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application session.commit() Sep 13 15:53:58.492451 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 1451, in commit Sep 13 15:53:58.492451 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application self._transaction.commit(_to_root=self.future) Sep 13 15:53:58.492451 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 829, in commit Sep 13 15:53:58.493309 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application self._prepare_impl() Sep 13 15:53:58.493309 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 808, in _prepare_impl Sep 13 15:53:58.493309 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application self.session.flush() Sep 13 15:53:58.493309 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3383, in flush Sep 13 15:53:58.493309 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application self._flush(objects) Sep 13 15:53:58.493309 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3523, in _flush Sep 13 15:53:58.493309 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application transaction.rollback(_capture_exception=True) Sep 13 15:53:58.493309 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__ Sep 13 15:53:58.493309 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application compat.raise_( Sep 13 15:53:58.493309 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Sep 13 15:53:58.493309 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application raise exception Sep 13 15:53:58.493309 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3483, in _flush Sep 13 15:53:58.493309 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application flush_context.execute() Sep 13 15:53:58.493309 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 456, in execute Sep 13 15:53:58.493309 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application rec.execute(self) Sep 13 15:53:58.493309 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 630, in execute Sep 13 15:53:58.493309 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application util.preloaded.orm_persistence.save_obj( Sep 13 15:53:58.494283 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 245, in save_obj Sep 13 15:53:58.494283 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application _emit_insert_statements( Sep 13 15:53:58.494283 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 1097, in _emit_insert_statements Sep 13 15:53:58.494283 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application c = connection._execute_20( Sep 13 15:53:58.494283 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1705, in _execute_20 Sep 13 15:53:58.494283 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return meth(self, args_10style, kwargs_10style, execution_options) Sep 13 15:53:58.494283 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/sql/elements.py", line 333, in _execute_on_connection Sep 13 15:53:58.494283 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return connection._execute_clauseelement( Sep 13 15:53:58.494283 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1572, in _execute_clauseelement Sep 13 15:53:58.494283 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application ret = self._execute_context( Sep 13 15:53:58.494283 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1943, in _execute_context Sep 13 15:53:58.494283 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application self._handle_dbapi_exception( Sep 13 15:53:58.494283 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 2122, in _handle_dbapi_exception Sep 13 15:53:58.494283 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application util.raise_(newraise, with_traceback=exc_info[2], from_=e) Sep 13 15:53:58.494283 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Sep 13 15:53:58.494283 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application raise exception Sep 13 15:53:58.495538 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Sep 13 15:53:58.495538 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application self.dialect.do_execute( Sep 13 15:53:58.495538 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Sep 13 15:53:58.495538 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Sep 13 15:53:58.495538 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Sep 13 15:53:58.495538 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application result = self._query(query) Sep 13 15:53:58.495538 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Sep 13 15:53:58.495538 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application conn.query(q) Sep 13 15:53:58.495538 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Sep 13 15:53:58.495538 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Sep 13 15:53:58.495538 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Sep 13 15:53:58.495538 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application result.read() Sep 13 15:53:58.495538 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Sep 13 15:53:58.495538 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Sep 13 15:53:58.495538 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Sep 13 15:53:58.495538 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application packet.raise_for_error() Sep 13 15:53:58.498102 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Sep 13 15:53:58.498102 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Sep 13 15:53:58.498102 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Sep 13 15:53:58.498102 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Sep 13 15:53:58.498102 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application oslo_db.exception.DBDuplicateEntry: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'anotherrole-<>' for key 'role.ixu_role_name_domain_id'") Sep 13 15:53:58.498102 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application [SQL: INSERT INTO `role` (id, name, description, extra) VALUES (%(id)s, %(name)s, %(description)s, %(extra)s)] Sep 13 15:53:58.498102 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application [parameters: {'id': '48de1c01d52a4586b0f45e0008983108', 'name': 'anotherrole', 'description': None, 'extra': '{}'}] Sep 13 15:53:58.498102 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application (Background on this error at: https://sqlalche.me/e/14/gkpj) Sep 13 15:53:58.498102 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application Sep 13 15:53:58.498102 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 13 15:53:58.498102 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application Sep 13 15:53:58.498102 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:53:58.498102 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:53:58.498102 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:53:58.498102 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:53:58.498102 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:53:58.500186 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:53:58.500186 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:53:58.500186 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:53:58.500186 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:53:58.500186 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:53:58.500186 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:53:58.500186 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 115, in post Sep 13 15:53:58.500186 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application ref = PROVIDERS.role_api.create_role( Sep 13 15:53:58.500186 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:53:58.500186 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:53:58.500186 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1298, in create_role Sep 13 15:53:58.500186 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application ret = self.driver.create_role(role_id, role) Sep 13 15:53:58.500186 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 563, in wrapper Sep 13 15:53:58.500186 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application raise exception.Conflict(type=conflict_type, Sep 13 15:53:58.500186 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application keystone.exception.Conflict: Conflict occurred attempting to store role - Duplicate entry found with name anotherrole. Sep 13 15:53:58.500186 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application Sep 13 15:53:58.502166 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d7bae79-29c9-48bc-b226-1b0676440b58 None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:53:58.504234 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 240/476] 10.222.0.50 () {60 vars in 1131 bytes} [Tue Sep 13 15:53:58 2022] POST /identity/v3/roles => generated 144 bytes in 263 msecs (HTTP/1.1 409) 5 headers in 183 bytes (2 switches on core 0) Sep 13 15:53:58.511307 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-085d20c3-0a9a-4fe0-a325-83f9eefff70d None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:53:58.565757 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f902c98a-9b5c-41e7-b3e8-09e5886a380c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:53:58.566762 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f902c98a-9b5c-41e7-b3e8-09e5886a380c None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:58.567270 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f902c98a-9b5c-41e7-b3e8-09e5886a380c None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:58.567740 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f902c98a-9b5c-41e7-b3e8-09e5886a380c None admin] PATH_INFO: `/v3/domains` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:58.568599 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f902c98a-9b5c-41e7-b3e8-09e5886a380c None admin] RBAC: Adding query filter params (name=Default) {{(pid=81438) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:53:58.569113 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f902c98a-9b5c-41e7-b3e8-09e5886a380c None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:53:58.570788 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f902c98a-9b5c-41e7-b3e8-09e5886a380c None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:53:58.584322 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:53:58.586698 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 237/477] 10.222.0.50 () {56 vars in 1109 bytes} [Tue Sep 13 15:53:58 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 268 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:53:58.594335 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f902c98a-9b5c-41e7-b3e8-09e5886a380c None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:53:58.715154 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f902c98a-9b5c-41e7-b3e8-09e5886a380c None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:53:58.723988 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:53:58.725762 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-085d20c3-0a9a-4fe0-a325-83f9eefff70d None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:53:58.768762 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47c0fe16-18ac-4313-a977-fa3d4f01bf73 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:53:58.769753 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47c0fe16-18ac-4313-a977-fa3d4f01bf73 None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:58.770263 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47c0fe16-18ac-4313-a977-fa3d4f01bf73 None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:58.770737 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47c0fe16-18ac-4313-a977-fa3d4f01bf73 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:58.775813 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-709383c3-de67-4744-afcf-23134244f835 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:53:58.776487 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-709383c3-de67-4744-afcf-23134244f835 None admin] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:58.776771 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-709383c3-de67-4744-afcf-23134244f835 None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:58.777109 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-709383c3-de67-4744-afcf-23134244f835 None admin] PATH_INFO: `/v3/users` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:58.777698 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-709383c3-de67-4744-afcf-23134244f835 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:53:58.779157 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-709383c3-de67-4744-afcf-23134244f835 None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:53:58.783417 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47c0fe16-18ac-4313-a977-fa3d4f01bf73 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 13 15:53:58.788725 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application [None req-47c0fe16-18ac-4313-a977-fa3d4f01bf73 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 13 15:53:58.788725 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:53:58.788725 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:53:58.788725 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:53:58.788725 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:53:58.788725 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:53:58.788725 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:53:58.788725 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:53:58.788725 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:53:58.788725 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:53:58.788725 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:53:58.788725 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:53:58.788725 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 13 15:53:58.788725 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 13 15:53:58.788725 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 13 15:53:58.788725 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 13 15:53:58.789604 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 13 15:53:58.789604 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 13 15:53:58.789604 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 13 15:53:58.789604 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 13 15:53:58.789604 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:53:58.789604 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:53:58.789604 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:53:58.789604 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:53:58.789604 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:53:58.789604 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:53:58.789604 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:53:58.789604 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application with Lock( Sep 13 15:53:58.789604 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:53:58.789604 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:53:58.789604 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:53:58.789604 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:53:58.790401 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:53:58.790401 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:53:58.790401 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:53:58.790401 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:53:58.790401 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 13 15:53:58.790401 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 13 15:53:58.790401 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 13 15:53:58.790401 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 13 15:53:58.790401 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 13 15:53:58.790401 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 13 15:53:58.790401 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 13 15:53:58.790401 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application Sep 13 15:53:58.793306 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 241/478] 10.222.0.50 () {56 vars in 1114 bytes} [Tue Sep 13 15:53:58 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 283 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:53:58.800223 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47c0fe16-18ac-4313-a977-fa3d4f01bf73 None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:53:58.816051 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 238/479] 10.222.0.50 () {60 vars in 1132 bytes} [Tue Sep 13 15:53:58 2022] POST /identity/v3/users => generated 249 bytes in 222 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 13 15:53:58.947080 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:53:58.955307 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47c0fe16-18ac-4313-a977-fa3d4f01bf73 None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:53:59.002151 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69e57016-c390-4d55-898d-fc2b77401383 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:53:59.002685 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69e57016-c390-4d55-898d-fc2b77401383 None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:53:59.002989 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69e57016-c390-4d55-898d-fc2b77401383 None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:53:59.003262 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69e57016-c390-4d55-898d-fc2b77401383 None admin] PATH_INFO: `/v3/roles` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:53:59.003932 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69e57016-c390-4d55-898d-fc2b77401383 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81437) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:53:59.004233 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69e57016-c390-4d55-898d-fc2b77401383 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:53:59.005734 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69e57016-c390-4d55-898d-fc2b77401383 None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:53:59.018229 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 242/480] 10.222.0.50 () {56 vars in 1111 bytes} [Tue Sep 13 15:53:58 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 218 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:54:00.053650 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8544cb57-fa12-452b-abce-9bd37dcd1b27 None None] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:00.054268 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8544cb57-fa12-452b-abce-9bd37dcd1b27 None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:00.054268 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8544cb57-fa12-452b-abce-9bd37dcd1b27 None None] PATH_INFO: `/` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:00.056097 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 239/481] 10.222.0.50 () {54 vars in 927 bytes} [Tue Sep 13 15:54:00 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:54:00.067960 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9db0d804-fac2-4962-95a0-c8a8ae71e3c6 None None] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:00.068367 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9db0d804-fac2-4962-95a0-c8a8ae71e3c6 None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:00.068586 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9db0d804-fac2-4962-95a0-c8a8ae71e3c6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:00.093961 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.auth.core [None req-9db0d804-fac2-4962-95a0-c8a8ae71e3c6 None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81437) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:54:00.095195 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-9db0d804-fac2-4962-95a0-c8a8ae71e3c6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:54:00.181905 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 243/482] 10.222.0.50 () {58 vars in 1024 bytes} [Tue Sep 13 15:54:00 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 115 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 13 15:54:00.581688 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8544cb57-fa12-452b-abce-9bd37dcd1b27 None None] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:00.589289 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71ba506d-4bd4-4b19-b492-b1645c11e59b None None] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:00.589577 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71ba506d-4bd4-4b19-b492-b1645c11e59b None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:00.589857 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71ba506d-4bd4-4b19-b492-b1645c11e59b None None] PATH_INFO: `/` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:00.591607 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 244/483] 10.222.0.50 () {54 vars in 927 bytes} [Tue Sep 13 15:54:00 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 13 15:54:00.596712 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ed8a751-74a3-4f55-a6d0-60da1f767739 None None] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:00.596994 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ed8a751-74a3-4f55-a6d0-60da1f767739 None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:00.597273 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ed8a751-74a3-4f55-a6d0-60da1f767739 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:00.616816 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.auth.core [None req-1ed8a751-74a3-4f55-a6d0-60da1f767739 None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81437) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:54:00.618008 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-1ed8a751-74a3-4f55-a6d0-60da1f767739 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:54:00.694067 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 245/484] 10.222.0.50 () {58 vars in 1024 bytes} [Tue Sep 13 15:54:00 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 13 15:54:00.695786 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7a1aa75-847a-403b-a52e-b9e9017cce89 None None] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:00.696267 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7a1aa75-847a-403b-a52e-b9e9017cce89 None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:00.696730 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7a1aa75-847a-403b-a52e-b9e9017cce89 None None] PATH_INFO: `/` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:00.698439 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 246/485] 10.222.0.50 () {54 vars in 927 bytes} [Tue Sep 13 15:54:00 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:54:00.701739 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8544cb57-fa12-452b-abce-9bd37dcd1b27 None None] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:00.703608 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-566fc51c-0015-47eb-ae59-1b5be6ec5e9e None None] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:00.704086 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-566fc51c-0015-47eb-ae59-1b5be6ec5e9e None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:00.704549 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-566fc51c-0015-47eb-ae59-1b5be6ec5e9e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:00.737722 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.auth.core [None req-566fc51c-0015-47eb-ae59-1b5be6ec5e9e None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81437) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:54:00.740623 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-566fc51c-0015-47eb-ae59-1b5be6ec5e9e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:54:00.760590 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:54:00.773058 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75684202-daf4-4639-ba20-44d9a2971c23 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:00.773687 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75684202-daf4-4639-ba20-44d9a2971c23 None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:00.773961 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75684202-daf4-4639-ba20-44d9a2971c23 None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:00.774227 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75684202-daf4-4639-ba20-44d9a2971c23 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:00.795665 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75684202-daf4-4639-ba20-44d9a2971c23 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 13 15:54:00.800804 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application [None req-75684202-daf4-4639-ba20-44d9a2971c23 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 13 15:54:00.800804 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:54:00.800804 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:54:00.800804 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:54:00.800804 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:54:00.800804 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:54:00.800804 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:54:00.800804 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:54:00.800804 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:54:00.800804 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:54:00.800804 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:54:00.800804 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:54:00.800804 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 13 15:54:00.800804 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 13 15:54:00.800804 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 13 15:54:00.800804 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 13 15:54:00.801719 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 13 15:54:00.801719 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 13 15:54:00.801719 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 13 15:54:00.801719 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 13 15:54:00.801719 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:54:00.801719 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:54:00.801719 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:54:00.801719 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:54:00.801719 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:54:00.801719 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:54:00.801719 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:54:00.801719 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application with Lock( Sep 13 15:54:00.801719 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:54:00.801719 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:54:00.801719 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:54:00.801719 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:54:00.803086 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:54:00.803086 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:54:00.803086 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:54:00.803086 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:54:00.803086 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 13 15:54:00.803086 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 13 15:54:00.803086 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 13 15:54:00.803086 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 13 15:54:00.803086 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 13 15:54:00.803086 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 13 15:54:00.803086 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 13 15:54:00.803086 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Sep 13 15:54:00.803086 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 240/486] 10.222.0.50 () {56 vars in 1102 bytes} [Tue Sep 13 15:54:00 2022] GET /identity/v3/roles/service => generated 85 bytes in 221 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:54:00.816275 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75684202-daf4-4639-ba20-44d9a2971c23 None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:00.861286 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:54:00.874319 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 247/487] 10.222.0.50 () {58 vars in 1024 bytes} [Tue Sep 13 15:54:00 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 171 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 13 15:54:00.974604 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75684202-daf4-4639-ba20-44d9a2971c23 None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:00.995909 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-566fc51c-0015-47eb-ae59-1b5be6ec5e9e None None] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:01.032503 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2300861a-e058-432a-ad0a-e188483bbee1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:01.033114 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2300861a-e058-432a-ad0a-e188483bbee1 None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:01.033390 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2300861a-e058-432a-ad0a-e188483bbee1 None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:01.033660 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2300861a-e058-432a-ad0a-e188483bbee1 None admin] PATH_INFO: `/v3/roles` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:01.034310 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2300861a-e058-432a-ad0a-e188483bbee1 None admin] RBAC: Adding query filter params (name=service) {{(pid=81438) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:54:01.034610 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2300861a-e058-432a-ad0a-e188483bbee1 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:01.036226 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2300861a-e058-432a-ad0a-e188483bbee1 None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:01.051978 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 241/488] 10.222.0.50 () {56 vars in 1103 bytes} [Tue Sep 13 15:54:00 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 236 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:54:01.060007 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2300861a-e058-432a-ad0a-e188483bbee1 None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:01.153954 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-566fc51c-0015-47eb-ae59-1b5be6ec5e9e None None] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:01.211236 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9dab8dc-d83b-4ef1-bcd3-1a76dc944ceb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:01.211877 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9dab8dc-d83b-4ef1-bcd3-1a76dc944ceb None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:01.212155 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9dab8dc-d83b-4ef1-bcd3-1a76dc944ceb None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:01.212428 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9dab8dc-d83b-4ef1-bcd3-1a76dc944ceb None admin] PATH_INFO: `/v3/roles/service` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:01.217386 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:54:01.225367 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9dab8dc-d83b-4ef1-bcd3-1a76dc944ceb None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 13 15:54:01.230780 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application [None req-f9dab8dc-d83b-4ef1-bcd3-1a76dc944ceb None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 13 15:54:01.230780 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:54:01.230780 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:54:01.230780 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:54:01.230780 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:54:01.230780 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:54:01.230780 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:54:01.230780 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:54:01.230780 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:54:01.230780 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:54:01.230780 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:54:01.230780 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:54:01.230780 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 13 15:54:01.230780 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 13 15:54:01.230780 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 13 15:54:01.230780 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 13 15:54:01.231729 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 13 15:54:01.231729 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 13 15:54:01.231729 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 13 15:54:01.231729 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 13 15:54:01.231729 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:54:01.231729 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:54:01.231729 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:54:01.231729 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:54:01.231729 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:54:01.231729 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:54:01.231729 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:54:01.231729 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application with Lock( Sep 13 15:54:01.231729 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:54:01.231729 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:54:01.231729 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:54:01.231729 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:54:01.232533 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:54:01.232533 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:54:01.232533 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:54:01.232533 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:54:01.232533 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 13 15:54:01.232533 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 13 15:54:01.232533 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 13 15:54:01.232533 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 13 15:54:01.232533 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 13 15:54:01.232533 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 13 15:54:01.232533 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 13 15:54:01.232533 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application Sep 13 15:54:01.232533 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 248/489] 10.222.0.50 () {56 vars in 1102 bytes} [Tue Sep 13 15:54:00 2022] GET /identity/v3/roles/service => generated 85 bytes in 237 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 13 15:54:01.233202 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9dab8dc-d83b-4ef1-bcd3-1a76dc944ceb None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:01.242468 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2300861a-e058-432a-ad0a-e188483bbee1 None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:01.243569 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:54:01.298012 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7acbfe03-69d9-47fb-b2b3-964de7f4aff7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:01.298780 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7acbfe03-69d9-47fb-b2b3-964de7f4aff7 None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:01.299098 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7acbfe03-69d9-47fb-b2b3-964de7f4aff7 None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:01.299641 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7acbfe03-69d9-47fb-b2b3-964de7f4aff7 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:01.316926 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7acbfe03-69d9-47fb-b2b3-964de7f4aff7 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:01.318391 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7acbfe03-69d9-47fb-b2b3-964de7f4aff7 None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:01.336872 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application [None req-7acbfe03-69d9-47fb-b2b3-964de7f4aff7 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:54:01.336872 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:54:01.336872 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:54:01.336872 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:54:01.336872 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:54:01.336872 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:54:01.336872 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:54:01.336872 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:54:01.336872 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:54:01.336872 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:54:01.336872 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:54:01.336872 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:54:01.336872 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 13 15:54:01.336872 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 13 15:54:01.336872 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 13 15:54:01.336872 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 13 15:54:01.338065 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:54:01.338065 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:54:01.338065 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:54:01.338065 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:54:01.338065 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:54:01.338065 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:54:01.338065 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:54:01.338065 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application with Lock( Sep 13 15:54:01.338065 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:54:01.338065 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:54:01.338065 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:54:01.338065 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:54:01.338065 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:54:01.338065 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:54:01.338065 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:54:01.338065 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:54:01.338065 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 13 15:54:01.339255 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 13 15:54:01.339255 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:54:01.339255 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Sep 13 15:54:01.345138 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 242/490] 10.222.0.50 () {56 vars in 1108 bytes} [Tue Sep 13 15:54:01 2022] GET /identity/v3/domains/Default => generated 87 bytes in 284 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:54:01.346006 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7acbfe03-69d9-47fb-b2b3-964de7f4aff7 None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:01.356719 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9dab8dc-d83b-4ef1-bcd3-1a76dc944ceb None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:01.392724 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b038c501-1d50-4346-af0e-31893ec22e6c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:01.393358 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b038c501-1d50-4346-af0e-31893ec22e6c None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:01.393637 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b038c501-1d50-4346-af0e-31893ec22e6c None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:01.393909 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b038c501-1d50-4346-af0e-31893ec22e6c None admin] PATH_INFO: `/v3/roles/service` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:01.404960 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b038c501-1d50-4346-af0e-31893ec22e6c None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 13 15:54:01.409736 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application [None req-b038c501-1d50-4346-af0e-31893ec22e6c None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 13 15:54:01.409736 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:54:01.409736 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:54:01.409736 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:54:01.409736 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:54:01.409736 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:54:01.409736 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:54:01.409736 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:54:01.409736 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:54:01.409736 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:54:01.409736 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:54:01.409736 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:54:01.409736 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 13 15:54:01.409736 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 13 15:54:01.409736 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 13 15:54:01.409736 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 13 15:54:01.411839 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 13 15:54:01.411839 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 13 15:54:01.411839 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 13 15:54:01.411839 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 13 15:54:01.411839 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:54:01.411839 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:54:01.411839 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:54:01.411839 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:54:01.411839 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:54:01.411839 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:54:01.411839 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:54:01.411839 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application with Lock( Sep 13 15:54:01.411839 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:54:01.411839 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:54:01.411839 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:54:01.411839 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:54:01.413950 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:54:01.413950 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:54:01.413950 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:54:01.413950 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:54:01.413950 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 13 15:54:01.413950 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 13 15:54:01.413950 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 13 15:54:01.413950 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 13 15:54:01.413950 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 13 15:54:01.413950 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 13 15:54:01.413950 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 13 15:54:01.413950 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application Sep 13 15:54:01.413950 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 249/491] 10.222.0.50 () {56 vars in 1102 bytes} [Tue Sep 13 15:54:01 2022] GET /identity/v3/roles/service => generated 85 bytes in 178 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:54:01.413950 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b038c501-1d50-4346-af0e-31893ec22e6c None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:01.479751 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7acbfe03-69d9-47fb-b2b3-964de7f4aff7 None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:01.481263 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:54:01.520713 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b038c501-1d50-4346-af0e-31893ec22e6c None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:01.523232 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ba2caa5-7e08-4b29-8970-57364d13e1f5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:01.524951 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ba2caa5-7e08-4b29-8970-57364d13e1f5 None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:01.525301 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ba2caa5-7e08-4b29-8970-57364d13e1f5 None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:01.526144 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ba2caa5-7e08-4b29-8970-57364d13e1f5 None admin] PATH_INFO: `/v3/roles` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:01.526990 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ba2caa5-7e08-4b29-8970-57364d13e1f5 None admin] RBAC: Adding query filter params (name=service) {{(pid=81438) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:54:01.527340 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ba2caa5-7e08-4b29-8970-57364d13e1f5 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:01.528803 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ba2caa5-7e08-4b29-8970-57364d13e1f5 None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:01.532418 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:54:01.546845 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 243/492] 10.222.0.50 () {56 vars in 1103 bytes} [Tue Sep 13 15:54:01 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 201 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:54:01.548311 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ba2caa5-7e08-4b29-8970-57364d13e1f5 None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:01.566535 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d5bc538-d58b-496b-97a3-74fa4c556dd3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:01.567351 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d5bc538-d58b-496b-97a3-74fa4c556dd3 None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:01.567847 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d5bc538-d58b-496b-97a3-74fa4c556dd3 None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:01.568247 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d5bc538-d58b-496b-97a3-74fa4c556dd3 None admin] PATH_INFO: `/v3/domains` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:01.569047 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d5bc538-d58b-496b-97a3-74fa4c556dd3 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81437) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:54:01.569452 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d5bc538-d58b-496b-97a3-74fa4c556dd3 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:01.571159 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d5bc538-d58b-496b-97a3-74fa4c556dd3 None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:01.584388 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 250/493] 10.222.0.50 () {56 vars in 1109 bytes} [Tue Sep 13 15:54:01 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:54:01.590868 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d5bc538-d58b-496b-97a3-74fa4c556dd3 None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:01.666423 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ba2caa5-7e08-4b29-8970-57364d13e1f5 None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:01.694388 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d5bc538-d58b-496b-97a3-74fa4c556dd3 None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:01.705782 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbaf9073-38ef-4659-a7fa-4350c5cc7f7e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:01.706592 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbaf9073-38ef-4659-a7fa-4350c5cc7f7e None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:01.707093 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbaf9073-38ef-4659-a7fa-4350c5cc7f7e None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:01.707539 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbaf9073-38ef-4659-a7fa-4350c5cc7f7e None admin] PATH_INFO: `/v3/roles` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:01.708505 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbaf9073-38ef-4659-a7fa-4350c5cc7f7e None admin] RBAC: Adding query filter params (name=service) {{(pid=81438) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:54:01.709007 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbaf9073-38ef-4659-a7fa-4350c5cc7f7e None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:01.710574 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbaf9073-38ef-4659-a7fa-4350c5cc7f7e None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:01.720358 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 244/494] 10.222.0.50 () {56 vars in 1103 bytes} [Tue Sep 13 15:54:01 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:54:01.720818 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:54:01.722023 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbaf9073-38ef-4659-a7fa-4350c5cc7f7e None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:01.734007 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa6e9913-add9-4618-9c32-0393f335d85f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:01.734646 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa6e9913-add9-4618-9c32-0393f335d85f None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:01.734951 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa6e9913-add9-4618-9c32-0393f335d85f None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:01.735230 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa6e9913-add9-4618-9c32-0393f335d85f None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:01.747162 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa6e9913-add9-4618-9c32-0393f335d85f None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:01.748627 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa6e9913-add9-4618-9c32-0393f335d85f None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:01.763238 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application [None req-aa6e9913-add9-4618-9c32-0393f335d85f None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:54:01.763238 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:54:01.763238 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:54:01.763238 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:54:01.763238 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:54:01.763238 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:54:01.763238 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:54:01.763238 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:54:01.763238 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:54:01.763238 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:54:01.763238 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:54:01.763238 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:54:01.763238 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 13 15:54:01.763238 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 13 15:54:01.763238 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 13 15:54:01.763238 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 13 15:54:01.764151 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:54:01.764151 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:54:01.764151 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:54:01.764151 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:54:01.764151 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:54:01.764151 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:54:01.764151 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:54:01.764151 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application with Lock( Sep 13 15:54:01.764151 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:54:01.764151 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:54:01.764151 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:54:01.764151 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:54:01.764151 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:54:01.764151 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:54:01.764151 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:54:01.764151 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:54:01.764151 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 13 15:54:01.765001 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 13 15:54:01.765001 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:54:01.765001 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application Sep 13 15:54:01.765001 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 251/495] 10.222.0.50 () {56 vars in 1108 bytes} [Tue Sep 13 15:54:01 2022] GET /identity/v3/domains/Default => generated 87 bytes in 179 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:54:01.765512 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ef6aa62-d868-4554-af87-1d1b98b65151 None None] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:01.765761 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ef6aa62-d868-4554-af87-1d1b98b65151 None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:01.766034 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ef6aa62-d868-4554-af87-1d1b98b65151 None None] PATH_INFO: `/` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:01.771022 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 252/496] 10.222.0.50 () {54 vars in 927 bytes} [Tue Sep 13 15:54:01 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:54:01.771022 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ef6aa62-d868-4554-af87-1d1b98b65151 None None] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:01.795603 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:54:01.844829 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbaf9073-38ef-4659-a7fa-4350c5cc7f7e None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:01.880769 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe197f01-00de-4281-9812-44ba9892b20a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:01.881404 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe197f01-00de-4281-9812-44ba9892b20a None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:01.881681 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe197f01-00de-4281-9812-44ba9892b20a None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:01.881959 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe197f01-00de-4281-9812-44ba9892b20a None admin] PATH_INFO: `/v3/users/nova` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:01.882327 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ef6aa62-d868-4554-af87-1d1b98b65151 None None] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:01.891685 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe197f01-00de-4281-9812-44ba9892b20a None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:01.893141 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe197f01-00de-4281-9812-44ba9892b20a None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:01.906633 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application [None req-fe197f01-00de-4281-9812-44ba9892b20a None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Sep 13 15:54:01.906633 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:54:01.906633 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:54:01.906633 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:54:01.906633 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:54:01.906633 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:54:01.906633 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:54:01.906633 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:54:01.906633 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:54:01.906633 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:54:01.906633 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:54:01.906633 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:54:01.906633 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 13 15:54:01.906633 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 13 15:54:01.906633 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 13 15:54:01.906633 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 13 15:54:01.907589 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:54:01.907589 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:54:01.907589 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 13 15:54:01.907589 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 13 15:54:01.907589 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 13 15:54:01.907589 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 13 15:54:01.907589 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:54:01.907589 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:54:01.907589 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:54:01.907589 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:54:01.907589 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:54:01.907589 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application with Lock( Sep 13 15:54:01.907589 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:54:01.907589 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:54:01.907589 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:54:01.907589 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:54:01.907589 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:54:01.907589 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:54:01.908499 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:54:01.908499 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:54:01.908499 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 13 15:54:01.908499 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 13 15:54:01.908499 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 13 15:54:01.908499 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 13 15:54:01.908499 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 13 15:54:01.908499 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 13 15:54:01.908499 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Sep 13 15:54:01.908499 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Sep 13 15:54:01.908499 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 245/497] 10.222.0.50 () {56 vars in 1093 bytes} [Tue Sep 13 15:54:01 2022] GET /identity/v3/users/nova => generated 82 bytes in 186 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:54:01.909086 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe197f01-00de-4281-9812-44ba9892b20a None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:01.924653 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2107bd1-4933-4f75-98c1-ebf5521f5997 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:01.925222 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2107bd1-4933-4f75-98c1-ebf5521f5997 None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:01.925520 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2107bd1-4933-4f75-98c1-ebf5521f5997 None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:01.925794 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2107bd1-4933-4f75-98c1-ebf5521f5997 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:01.936726 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2107bd1-4933-4f75-98c1-ebf5521f5997 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:01.938191 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2107bd1-4933-4f75-98c1-ebf5521f5997 None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:01.953115 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application [None req-d2107bd1-4933-4f75-98c1-ebf5521f5997 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:54:01.953115 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:54:01.953115 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:54:01.953115 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:54:01.953115 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:54:01.953115 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:54:01.953115 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:54:01.953115 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:54:01.953115 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:54:01.953115 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:54:01.953115 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:54:01.953115 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:54:01.953115 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 13 15:54:01.953115 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 13 15:54:01.953115 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 13 15:54:01.953115 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 13 15:54:01.954104 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:54:01.954104 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:54:01.954104 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:54:01.954104 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:54:01.954104 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:54:01.954104 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:54:01.954104 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:54:01.954104 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application with Lock( Sep 13 15:54:01.954104 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:54:01.954104 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:54:01.954104 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:54:01.954104 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:54:01.954104 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:54:01.954104 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:54:01.954104 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:54:01.954104 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:54:01.954104 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 13 15:54:01.955011 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 13 15:54:01.955011 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:54:01.955011 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application Sep 13 15:54:01.955011 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 253/498] 10.222.0.50 () {56 vars in 1108 bytes} [Tue Sep 13 15:54:01 2022] GET /identity/v3/domains/Default => generated 87 bytes in 186 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:54:01.955623 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78a98ff8-aacb-4e05-ba3b-c2a8659b8bf2 None None] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:01.955799 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78a98ff8-aacb-4e05-ba3b-c2a8659b8bf2 None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:01.956084 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78a98ff8-aacb-4e05-ba3b-c2a8659b8bf2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:01.973529 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.auth.core [None req-78a98ff8-aacb-4e05-ba3b-c2a8659b8bf2 None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81437) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:54:01.974727 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-78a98ff8-aacb-4e05-ba3b-c2a8659b8bf2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:54:01.998264 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:54:02.013550 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe197f01-00de-4281-9812-44ba9892b20a None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:02.034202 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:54:02.042404 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 254/499] 10.222.0.50 () {58 vars in 1024 bytes} [Tue Sep 13 15:54:01 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 13 15:54:02.043279 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78a98ff8-aacb-4e05-ba3b-c2a8659b8bf2 None None] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:02.065038 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-736cba45-0003-42eb-8e70-0646d3443491 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:02.065998 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-736cba45-0003-42eb-8e70-0646d3443491 None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:02.066351 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-736cba45-0003-42eb-8e70-0646d3443491 None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:02.066732 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-736cba45-0003-42eb-8e70-0646d3443491 None admin] PATH_INFO: `/v3/domains` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:02.067555 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-736cba45-0003-42eb-8e70-0646d3443491 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81438) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:54:02.068013 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-736cba45-0003-42eb-8e70-0646d3443491 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:02.069476 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-736cba45-0003-42eb-8e70-0646d3443491 None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:02.083363 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 246/500] 10.222.0.50 () {56 vars in 1109 bytes} [Tue Sep 13 15:54:01 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:54:02.084646 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a7e09b2-8455-4d29-8afa-9a8b7aa6244d None None] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:02.084940 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a7e09b2-8455-4d29-8afa-9a8b7aa6244d None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:02.085215 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a7e09b2-8455-4d29-8afa-9a8b7aa6244d None None] PATH_INFO: `/` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:02.086557 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 247/501] 10.222.0.50 () {54 vars in 927 bytes} [Tue Sep 13 15:54:02 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:54:02.087267 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a7e09b2-8455-4d29-8afa-9a8b7aa6244d None None] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:02.156955 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78a98ff8-aacb-4e05-ba3b-c2a8659b8bf2 None None] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:02.191623 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5a077fb-4fca-4e55-8c21-e36f17206d74 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:02.192153 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5a077fb-4fca-4e55-8c21-e36f17206d74 None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:02.192431 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5a077fb-4fca-4e55-8c21-e36f17206d74 None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:02.192700 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5a077fb-4fca-4e55-8c21-e36f17206d74 None admin] PATH_INFO: `/v3/users` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:02.193418 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5a077fb-4fca-4e55-8c21-e36f17206d74 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81437) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:54:02.193725 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5a077fb-4fca-4e55-8c21-e36f17206d74 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:02.194763 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a7e09b2-8455-4d29-8afa-9a8b7aa6244d None None] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:02.195186 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5a077fb-4fca-4e55-8c21-e36f17206d74 None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:02.230128 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b03399d-90bb-4207-baa0-75a6892178f9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:02.230730 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b03399d-90bb-4207-baa0-75a6892178f9 None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:02.231087 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b03399d-90bb-4207-baa0-75a6892178f9 None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:02.231361 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b03399d-90bb-4207-baa0-75a6892178f9 None admin] PATH_INFO: `/v3/domains` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:02.236930 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 255/502] 10.222.0.50 () {56 vars in 1133 bytes} [Tue Sep 13 15:54:02 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:54:02.237065 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b03399d-90bb-4207-baa0-75a6892178f9 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81438) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:54:02.237065 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b03399d-90bb-4207-baa0-75a6892178f9 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:02.237240 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5a077fb-4fca-4e55-8c21-e36f17206d74 None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:02.237322 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b03399d-90bb-4207-baa0-75a6892178f9 None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:02.252383 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 248/503] 10.222.0.50 () {56 vars in 1109 bytes} [Tue Sep 13 15:54:02 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:54:02.252383 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcaac2f2-119d-4cea-ac4e-0051bf6f11e8 None None] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:02.252383 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcaac2f2-119d-4cea-ac4e-0051bf6f11e8 None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:02.252383 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcaac2f2-119d-4cea-ac4e-0051bf6f11e8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:02.257873 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:54:02.270481 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.auth.core [None req-bcaac2f2-119d-4cea-ac4e-0051bf6f11e8 None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81438) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:54:02.271702 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-bcaac2f2-119d-4cea-ac4e-0051bf6f11e8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:54:02.307343 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:54:02.342007 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5a077fb-4fca-4e55-8c21-e36f17206d74 None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:02.342223 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 249/504] 10.222.0.50 () {58 vars in 1024 bytes} [Tue Sep 13 15:54:02 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 13 15:54:02.342979 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcaac2f2-119d-4cea-ac4e-0051bf6f11e8 None None] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:02.376542 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60a204a1-a890-4ba0-9a4b-f3ead319465d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:02.377403 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60a204a1-a890-4ba0-9a4b-f3ead319465d None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:02.377843 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60a204a1-a890-4ba0-9a4b-f3ead319465d None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:02.378125 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60a204a1-a890-4ba0-9a4b-f3ead319465d None admin] PATH_INFO: `/v3/users/cinder` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:02.386313 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60a204a1-a890-4ba0-9a4b-f3ead319465d None admin] RBAC: Authorizing `identity:get_user(user_id=cinder)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:02.387811 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60a204a1-a890-4ba0-9a4b-f3ead319465d None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:02.400321 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application [None req-60a204a1-a890-4ba0-9a4b-f3ead319465d None admin] Could not find user: cinder.: keystone.exception.UserNotFound: Could not find user: cinder. Sep 13 15:54:02.400321 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:54:02.400321 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:54:02.400321 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:54:02.400321 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:54:02.400321 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:54:02.400321 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:54:02.400321 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:54:02.400321 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:54:02.400321 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:54:02.400321 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:54:02.400321 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:54:02.400321 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 13 15:54:02.400321 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 13 15:54:02.400321 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 13 15:54:02.400321 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 13 15:54:02.401923 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:54:02.401923 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:54:02.401923 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 13 15:54:02.401923 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 13 15:54:02.401923 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 13 15:54:02.401923 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 13 15:54:02.401923 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:54:02.401923 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:54:02.401923 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:54:02.401923 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:54:02.401923 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:54:02.401923 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application with Lock( Sep 13 15:54:02.401923 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:54:02.401923 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:54:02.401923 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:54:02.401923 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:54:02.401923 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:54:02.401923 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:54:02.402840 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:54:02.402840 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:54:02.402840 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 13 15:54:02.402840 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 13 15:54:02.402840 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 13 15:54:02.402840 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 13 15:54:02.402840 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 13 15:54:02.402840 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 13 15:54:02.402840 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: cinder. Sep 13 15:54:02.402840 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application Sep 13 15:54:02.402840 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 256/505] 10.222.0.50 () {56 vars in 1099 bytes} [Tue Sep 13 15:54:02 2022] GET /identity/v3/users/cinder => generated 84 bytes in 170 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:54:02.403443 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60a204a1-a890-4ba0-9a4b-f3ead319465d None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:02.460336 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcaac2f2-119d-4cea-ac4e-0051bf6f11e8 None None] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:02.490785 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:54:02.497520 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bad7547-c030-49f3-91e9-ea5a91bfb47c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:02.498133 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bad7547-c030-49f3-91e9-ea5a91bfb47c None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:02.498418 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bad7547-c030-49f3-91e9-ea5a91bfb47c None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:02.498693 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bad7547-c030-49f3-91e9-ea5a91bfb47c None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:02.516806 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60a204a1-a890-4ba0-9a4b-f3ead319465d None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:02.539471 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:54:02.551896 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b54a650a-3707-44af-97b5-c7885cc454e9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:02.552768 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b54a650a-3707-44af-97b5-c7885cc454e9 None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:02.553240 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b54a650a-3707-44af-97b5-c7885cc454e9 None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:02.553513 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b54a650a-3707-44af-97b5-c7885cc454e9 None admin] PATH_INFO: `/v3/users/glance` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:02.561300 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b54a650a-3707-44af-97b5-c7885cc454e9 None admin] RBAC: Authorizing `identity:get_user(user_id=glance)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:02.562760 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b54a650a-3707-44af-97b5-c7885cc454e9 None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:02.572685 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8bad7547-c030-49f3-91e9-ea5a91bfb47c None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:02.574106 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application [None req-b54a650a-3707-44af-97b5-c7885cc454e9 None admin] Could not find user: glance.: keystone.exception.UserNotFound: Could not find user: glance. Sep 13 15:54:02.574106 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:54:02.574106 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:54:02.574106 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:54:02.574106 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:54:02.574106 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:54:02.574106 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:54:02.574106 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:54:02.574106 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:54:02.574106 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:54:02.574106 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:54:02.574106 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:54:02.574106 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 13 15:54:02.574106 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 13 15:54:02.574106 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 13 15:54:02.574106 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 13 15:54:02.575034 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:54:02.575034 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:54:02.575034 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 13 15:54:02.575034 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 13 15:54:02.575034 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 13 15:54:02.575034 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 13 15:54:02.575034 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:54:02.575034 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:54:02.575034 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:54:02.575034 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:54:02.575034 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:54:02.575034 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application with Lock( Sep 13 15:54:02.575034 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:54:02.575034 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:54:02.575034 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:54:02.575034 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:54:02.575034 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:54:02.575034 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:54:02.575941 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:54:02.575941 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:54:02.575941 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 13 15:54:02.575941 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 13 15:54:02.575941 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 13 15:54:02.575941 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 13 15:54:02.575941 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 13 15:54:02.575941 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 13 15:54:02.575941 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance. Sep 13 15:54:02.575941 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application Sep 13 15:54:02.576446 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8bad7547-c030-49f3-91e9-ea5a91bfb47c None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:02.580005 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 257/506] 10.222.0.50 () {56 vars in 1099 bytes} [Tue Sep 13 15:54:02 2022] GET /identity/v3/users/glance => generated 84 bytes in 177 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:54:02.580734 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b54a650a-3707-44af-97b5-c7885cc454e9 None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:02.675580 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 250/507] 10.222.0.50 () {58 vars in 1282 bytes} [Tue Sep 13 15:54:02 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 333 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 13 15:54:02.676599 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bad7547-c030-49f3-91e9-ea5a91bfb47c None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:02.679565 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b54a650a-3707-44af-97b5-c7885cc454e9 None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:02.713455 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1466b40-c84c-468e-88c0-2cdbf295b28a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:02.714162 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1466b40-c84c-468e-88c0-2cdbf295b28a None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:02.714476 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1466b40-c84c-468e-88c0-2cdbf295b28a None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:02.714750 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1466b40-c84c-468e-88c0-2cdbf295b28a None admin] PATH_INFO: `/v3/roles/service` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:02.721527 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:54:02.726307 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1466b40-c84c-468e-88c0-2cdbf295b28a None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 13 15:54:02.732190 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application [None req-b1466b40-c84c-468e-88c0-2cdbf295b28a None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 13 15:54:02.732190 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:54:02.732190 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:54:02.732190 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:54:02.732190 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:54:02.732190 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:54:02.732190 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:54:02.732190 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:54:02.732190 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:54:02.732190 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:54:02.732190 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:54:02.732190 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:54:02.732190 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 13 15:54:02.732190 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 13 15:54:02.732190 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 13 15:54:02.732190 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 13 15:54:02.733123 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 13 15:54:02.733123 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 13 15:54:02.733123 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 13 15:54:02.733123 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 13 15:54:02.733123 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:54:02.733123 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:54:02.733123 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:54:02.733123 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:54:02.733123 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:54:02.733123 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:54:02.733123 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:54:02.733123 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application with Lock( Sep 13 15:54:02.733123 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:54:02.733123 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:54:02.733123 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:54:02.733123 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:54:02.733927 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:54:02.733927 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:54:02.733927 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:54:02.733927 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:54:02.733927 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 13 15:54:02.733927 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 13 15:54:02.733927 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 13 15:54:02.733927 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 13 15:54:02.733927 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 13 15:54:02.733927 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 13 15:54:02.733927 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 13 15:54:02.733927 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application Sep 13 15:54:02.733927 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 258/508] 10.222.0.50 () {56 vars in 1102 bytes} [Tue Sep 13 15:54:02 2022] GET /identity/v3/roles/service => generated 85 bytes in 153 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:54:02.734764 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1466b40-c84c-468e-88c0-2cdbf295b28a None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:02.751203 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:54:02.782832 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bad7547-c030-49f3-91e9-ea5a91bfb47c None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:02.816855 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0119639-2a43-4abb-8c13-8481cef7e07e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:02.817457 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0119639-2a43-4abb-8c13-8481cef7e07e None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:02.817734 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0119639-2a43-4abb-8c13-8481cef7e07e None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:02.818005 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0119639-2a43-4abb-8c13-8481cef7e07e None admin] PATH_INFO: `/v3/users` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:02.818719 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0119639-2a43-4abb-8c13-8481cef7e07e None admin] RBAC: Adding query filter params (domain_id=default, name=cinder) {{(pid=81438) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:54:02.819055 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0119639-2a43-4abb-8c13-8481cef7e07e None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:02.820501 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0119639-2a43-4abb-8c13-8481cef7e07e None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:02.833528 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1466b40-c84c-468e-88c0-2cdbf295b28a None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:02.856752 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 251/509] 10.222.0.50 () {56 vars in 1137 bytes} [Tue Sep 13 15:54:02 2022] GET /identity/v3/users?domain_id=default&name=cinder => generated 372 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:54:02.857581 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0119639-2a43-4abb-8c13-8481cef7e07e None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:02.870605 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e50070e-1e3c-4d89-8834-e348a7db1fca None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:02.871242 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e50070e-1e3c-4d89-8834-e348a7db1fca None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:02.871519 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e50070e-1e3c-4d89-8834-e348a7db1fca None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:02.871821 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e50070e-1e3c-4d89-8834-e348a7db1fca None admin] PATH_INFO: `/v3/users` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:02.872542 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e50070e-1e3c-4d89-8834-e348a7db1fca None admin] RBAC: Adding query filter params (domain_id=default, name=glance) {{(pid=81437) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:54:02.872849 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e50070e-1e3c-4d89-8834-e348a7db1fca None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:02.874293 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e50070e-1e3c-4d89-8834-e348a7db1fca None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:02.891559 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 259/510] 10.222.0.50 () {56 vars in 1137 bytes} [Tue Sep 13 15:54:02 2022] GET /identity/v3/users?domain_id=default&name=glance => generated 372 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:54:02.892349 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e50070e-1e3c-4d89-8834-e348a7db1fca None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:02.946631 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:54:02.956082 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0119639-2a43-4abb-8c13-8481cef7e07e None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:02.983055 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:54:02.991206 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c8de0a2-7ca0-4907-9ce3-4b040f030ab0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:02.991405 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e50070e-1e3c-4d89-8834-e348a7db1fca None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:02.991840 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c8de0a2-7ca0-4907-9ce3-4b040f030ab0 None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:02.992120 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c8de0a2-7ca0-4907-9ce3-4b040f030ab0 None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:02.992393 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c8de0a2-7ca0-4907-9ce3-4b040f030ab0 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:03.002467 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c8de0a2-7ca0-4907-9ce3-4b040f030ab0 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:03.003977 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c8de0a2-7ca0-4907-9ce3-4b040f030ab0 None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:03.019015 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application [None req-9c8de0a2-7ca0-4907-9ce3-4b040f030ab0 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:54:03.019015 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:54:03.019015 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:54:03.019015 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:54:03.019015 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:54:03.019015 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:54:03.019015 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:54:03.019015 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:54:03.019015 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:54:03.019015 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:54:03.019015 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:54:03.019015 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:54:03.019015 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 13 15:54:03.019015 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 13 15:54:03.019015 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 13 15:54:03.019015 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 13 15:54:03.019961 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:54:03.019961 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:54:03.019961 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:54:03.019961 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:54:03.019961 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:54:03.019961 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:54:03.019961 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:54:03.019961 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application with Lock( Sep 13 15:54:03.019961 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:54:03.019961 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:54:03.019961 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:54:03.019961 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:54:03.019961 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:54:03.019961 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:54:03.019961 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:54:03.019961 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:54:03.019961 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 13 15:54:03.020893 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 13 15:54:03.020893 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:54:03.020893 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Sep 13 15:54:03.025056 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 252/511] 10.222.0.50 () {56 vars in 1108 bytes} [Tue Sep 13 15:54:02 2022] GET /identity/v3/domains/Default => generated 87 bytes in 168 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:54:03.026470 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c8de0a2-7ca0-4907-9ce3-4b040f030ab0 None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:03.026869 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c7447c9-a5cb-476b-b9e7-f7e9f80f8ba9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:03.027522 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c7447c9-a5cb-476b-b9e7-f7e9f80f8ba9 None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:03.027799 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c7447c9-a5cb-476b-b9e7-f7e9f80f8ba9 None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:03.028071 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c7447c9-a5cb-476b-b9e7-f7e9f80f8ba9 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:03.038087 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c7447c9-a5cb-476b-b9e7-f7e9f80f8ba9 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:03.039866 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c7447c9-a5cb-476b-b9e7-f7e9f80f8ba9 None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:03.055355 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application [None req-2c7447c9-a5cb-476b-b9e7-f7e9f80f8ba9 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:54:03.055355 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:54:03.055355 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:54:03.055355 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:54:03.055355 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:54:03.055355 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:54:03.055355 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:54:03.055355 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:54:03.055355 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:54:03.055355 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:54:03.055355 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:54:03.055355 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:54:03.055355 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 13 15:54:03.055355 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 13 15:54:03.055355 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 13 15:54:03.055355 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 13 15:54:03.056318 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:54:03.056318 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:54:03.056318 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:54:03.056318 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:54:03.056318 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:54:03.056318 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:54:03.056318 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:54:03.056318 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application with Lock( Sep 13 15:54:03.056318 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:54:03.056318 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:54:03.056318 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:54:03.056318 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:54:03.056318 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:54:03.056318 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:54:03.056318 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:54:03.056318 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:54:03.056318 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 13 15:54:03.057308 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 13 15:54:03.057308 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:54:03.057308 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application Sep 13 15:54:03.057308 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 260/512] 10.222.0.50 () {56 vars in 1108 bytes} [Tue Sep 13 15:54:02 2022] GET /identity/v3/domains/Default => generated 87 bytes in 164 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:54:03.057529 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c7447c9-a5cb-476b-b9e7-f7e9f80f8ba9 None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:03.130638 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c8de0a2-7ca0-4907-9ce3-4b040f030ab0 None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:03.157400 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c7447c9-a5cb-476b-b9e7-f7e9f80f8ba9 None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:03.164630 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3fd6678-dd4e-48db-8241-dac86d4a9a02 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:03.165230 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3fd6678-dd4e-48db-8241-dac86d4a9a02 None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:03.165510 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3fd6678-dd4e-48db-8241-dac86d4a9a02 None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:03.165783 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3fd6678-dd4e-48db-8241-dac86d4a9a02 None admin] PATH_INFO: `/v3/roles` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:03.166477 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:54:03.166697 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3fd6678-dd4e-48db-8241-dac86d4a9a02 None admin] RBAC: Adding query filter params (name=service) {{(pid=81438) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:54:03.166778 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3fd6678-dd4e-48db-8241-dac86d4a9a02 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:03.168173 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3fd6678-dd4e-48db-8241-dac86d4a9a02 None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:03.175983 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 253/513] 10.222.0.50 () {56 vars in 1103 bytes} [Tue Sep 13 15:54:03 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:54:03.176769 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3fd6678-dd4e-48db-8241-dac86d4a9a02 None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:03.193424 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47ac63a6-44f8-482d-b9b4-2d4dc262690c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:03.194037 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47ac63a6-44f8-482d-b9b4-2d4dc262690c None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:03.194317 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47ac63a6-44f8-482d-b9b4-2d4dc262690c None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:03.194588 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47ac63a6-44f8-482d-b9b4-2d4dc262690c None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:03.220446 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:54:03.262781 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47ac63a6-44f8-482d-b9b4-2d4dc262690c None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:03.264188 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47ac63a6-44f8-482d-b9b4-2d4dc262690c None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:03.278205 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3fd6678-dd4e-48db-8241-dac86d4a9a02 None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:03.312728 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26217427-75ff-474d-aeb6-2fe4a3f165e5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:03.313335 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26217427-75ff-474d-aeb6-2fe4a3f165e5 None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:03.313619 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26217427-75ff-474d-aeb6-2fe4a3f165e5 None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:03.313896 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26217427-75ff-474d-aeb6-2fe4a3f165e5 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:03.361113 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 261/514] 10.222.0.50 () {58 vars in 1282 bytes} [Tue Sep 13 15:54:03 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 304 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 13 15:54:03.361955 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47ac63a6-44f8-482d-b9b4-2d4dc262690c None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:03.372020 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:54:03.382228 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26217427-75ff-474d-aeb6-2fe4a3f165e5 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:03.383698 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26217427-75ff-474d-aeb6-2fe4a3f165e5 None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:03.425798 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:54:03.458442 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47ac63a6-44f8-482d-b9b4-2d4dc262690c None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:03.484509 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 254/515] 10.222.0.50 () {58 vars in 1282 bytes} [Tue Sep 13 15:54:03 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 308 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 13 15:54:03.485344 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26217427-75ff-474d-aeb6-2fe4a3f165e5 None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:03.494637 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04cf2d7e-0b08-4c9e-84ba-64a29b822ec3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:03.495376 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04cf2d7e-0b08-4c9e-84ba-64a29b822ec3 None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:03.495659 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04cf2d7e-0b08-4c9e-84ba-64a29b822ec3 None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:03.495934 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04cf2d7e-0b08-4c9e-84ba-64a29b822ec3 None admin] PATH_INFO: `/v3/domains` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:03.496572 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04cf2d7e-0b08-4c9e-84ba-64a29b822ec3 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81437) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:54:03.496877 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04cf2d7e-0b08-4c9e-84ba-64a29b822ec3 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:03.498264 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04cf2d7e-0b08-4c9e-84ba-64a29b822ec3 None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:03.509188 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 262/516] 10.222.0.50 () {56 vars in 1109 bytes} [Tue Sep 13 15:54:03 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:54:03.510228 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04cf2d7e-0b08-4c9e-84ba-64a29b822ec3 None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:03.574965 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:54:03.588486 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26217427-75ff-474d-aeb6-2fe4a3f165e5 None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:03.608547 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04cf2d7e-0b08-4c9e-84ba-64a29b822ec3 None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:03.624292 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f4be8e2-03b1-47e7-80da-e76d0379ae04 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:03.624895 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f4be8e2-03b1-47e7-80da-e76d0379ae04 None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:03.625175 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f4be8e2-03b1-47e7-80da-e76d0379ae04 None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:03.625446 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f4be8e2-03b1-47e7-80da-e76d0379ae04 None admin] PATH_INFO: `/v3/domains` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:03.626078 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f4be8e2-03b1-47e7-80da-e76d0379ae04 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81438) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:54:03.626378 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f4be8e2-03b1-47e7-80da-e76d0379ae04 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:03.627786 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f4be8e2-03b1-47e7-80da-e76d0379ae04 None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:03.639463 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 255/517] 10.222.0.50 () {56 vars in 1109 bytes} [Tue Sep 13 15:54:03 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:54:03.640252 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f4be8e2-03b1-47e7-80da-e76d0379ae04 None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:03.645842 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-713f9dca-326c-4fc4-86e4-6b47309712c4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:03.646122 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:54:03.646797 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-713f9dca-326c-4fc4-86e4-6b47309712c4 None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:03.646797 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-713f9dca-326c-4fc4-86e4-6b47309712c4 None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:03.647114 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-713f9dca-326c-4fc4-86e4-6b47309712c4 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:03.658612 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-713f9dca-326c-4fc4-86e4-6b47309712c4 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:03.660103 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-713f9dca-326c-4fc4-86e4-6b47309712c4 None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:03.674091 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application [None req-713f9dca-326c-4fc4-86e4-6b47309712c4 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:54:03.674091 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:54:03.674091 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:54:03.674091 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:54:03.674091 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:54:03.674091 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:54:03.674091 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:54:03.674091 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:54:03.674091 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:54:03.674091 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:54:03.674091 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:54:03.674091 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:54:03.674091 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 13 15:54:03.674091 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 13 15:54:03.674091 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 13 15:54:03.674091 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 13 15:54:03.674992 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:54:03.674992 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:54:03.674992 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:54:03.674992 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:54:03.674992 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:54:03.674992 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:54:03.674992 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:54:03.674992 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application with Lock( Sep 13 15:54:03.674992 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:54:03.674992 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:54:03.674992 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:54:03.674992 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:54:03.674992 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:54:03.674992 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:54:03.674992 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:54:03.674992 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:54:03.674992 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 13 15:54:03.675854 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 13 15:54:03.675854 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:54:03.675854 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application Sep 13 15:54:03.675854 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 263/518] 10.222.0.50 () {56 vars in 1108 bytes} [Tue Sep 13 15:54:03 2022] GET /identity/v3/domains/Default => generated 87 bytes in 165 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:54:03.676072 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-713f9dca-326c-4fc4-86e4-6b47309712c4 None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:03.747129 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f4be8e2-03b1-47e7-80da-e76d0379ae04 None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:03.775727 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-713f9dca-326c-4fc4-86e4-6b47309712c4 None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:03.781083 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8162bd1-c08a-41b0-80dc-67b6ca8ba83f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:03.781715 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8162bd1-c08a-41b0-80dc-67b6ca8ba83f None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:03.781998 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8162bd1-c08a-41b0-80dc-67b6ca8ba83f None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:03.782276 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8162bd1-c08a-41b0-80dc-67b6ca8ba83f None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:03.793321 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8162bd1-c08a-41b0-80dc-67b6ca8ba83f None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:03.794773 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8162bd1-c08a-41b0-80dc-67b6ca8ba83f None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:03.795571 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:54:03.809881 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application [None req-a8162bd1-c08a-41b0-80dc-67b6ca8ba83f None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:54:03.809881 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:54:03.809881 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:54:03.809881 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:54:03.809881 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:54:03.809881 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:54:03.809881 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:54:03.809881 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:54:03.809881 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:54:03.809881 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:54:03.809881 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:54:03.809881 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:54:03.809881 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 13 15:54:03.809881 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 13 15:54:03.809881 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 13 15:54:03.809881 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 13 15:54:03.811005 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:54:03.811005 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:54:03.811005 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:54:03.811005 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:54:03.811005 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:54:03.811005 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:54:03.811005 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:54:03.811005 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application with Lock( Sep 13 15:54:03.811005 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:54:03.811005 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:54:03.811005 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:54:03.811005 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:54:03.811005 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:54:03.811005 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:54:03.811005 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:54:03.811005 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:54:03.811005 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 13 15:54:03.812075 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 13 15:54:03.812075 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:54:03.812075 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Sep 13 15:54:03.812594 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 256/519] 10.222.0.50 () {56 vars in 1108 bytes} [Tue Sep 13 15:54:03 2022] GET /identity/v3/domains/Default => generated 87 bytes in 173 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:54:03.812945 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8b46078-83c9-45b3-bfd8-e8660324006a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:03.813874 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8162bd1-c08a-41b0-80dc-67b6ca8ba83f None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:03.813995 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8b46078-83c9-45b3-bfd8-e8660324006a None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:03.813995 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8b46078-83c9-45b3-bfd8-e8660324006a None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:03.814175 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8b46078-83c9-45b3-bfd8-e8660324006a None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:03.828533 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8b46078-83c9-45b3-bfd8-e8660324006a None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:03.829923 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8b46078-83c9-45b3-bfd8-e8660324006a None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:03.844166 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application [None req-d8b46078-83c9-45b3-bfd8-e8660324006a None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:54:03.844166 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:54:03.844166 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:54:03.844166 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:54:03.844166 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:54:03.844166 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:54:03.844166 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:54:03.844166 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:54:03.844166 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:54:03.844166 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:54:03.844166 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:54:03.844166 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:54:03.844166 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 13 15:54:03.844166 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 13 15:54:03.844166 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 13 15:54:03.844166 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 13 15:54:03.845083 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:54:03.845083 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:54:03.845083 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:54:03.845083 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:54:03.845083 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:54:03.845083 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:54:03.845083 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:54:03.845083 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application with Lock( Sep 13 15:54:03.845083 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:54:03.845083 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:54:03.845083 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:54:03.845083 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:54:03.845083 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:54:03.845083 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:54:03.845083 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:54:03.845083 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:54:03.845083 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 13 15:54:03.845930 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 13 15:54:03.845930 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:54:03.845930 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application Sep 13 15:54:03.845930 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 264/520] 10.222.0.50 () {56 vars in 1108 bytes} [Tue Sep 13 15:54:03 2022] GET /identity/v3/domains/Default => generated 87 bytes in 169 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:54:03.846156 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8b46078-83c9-45b3-bfd8-e8660324006a None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:03.875745 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:54:03.921770 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8162bd1-c08a-41b0-80dc-67b6ca8ba83f None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:03.944766 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8b46078-83c9-45b3-bfd8-e8660324006a None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:03.956118 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9b07581-3967-4ab5-86cf-767d101fe78e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:03.956724 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9b07581-3967-4ab5-86cf-767d101fe78e None admin] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:03.957029 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9b07581-3967-4ab5-86cf-767d101fe78e None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:03.957284 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9b07581-3967-4ab5-86cf-767d101fe78e None admin] PATH_INFO: `/v3/users` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:03.957924 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9b07581-3967-4ab5-86cf-767d101fe78e None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:03.959394 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9b07581-3967-4ab5-86cf-767d101fe78e None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:03.979469 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db7d388c-47aa-4acd-8035-f111ba30483d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:03.980068 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db7d388c-47aa-4acd-8035-f111ba30483d None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:03.980344 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db7d388c-47aa-4acd-8035-f111ba30483d None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:03.980616 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db7d388c-47aa-4acd-8035-f111ba30483d None admin] PATH_INFO: `/v3/projects/service` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:03.982824 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 257/521] 10.222.0.50 () {60 vars in 1132 bytes} [Tue Sep 13 15:54:03 2022] POST /identity/v3/users => generated 247 bytes in 170 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 15:54:03.983713 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9b07581-3967-4ab5-86cf-767d101fe78e None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:03.987492 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db7d388c-47aa-4acd-8035-f111ba30483d None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:03.989002 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db7d388c-47aa-4acd-8035-f111ba30483d None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:04.002348 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application [None req-db7d388c-47aa-4acd-8035-f111ba30483d None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 13 15:54:04.002348 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:54:04.002348 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:54:04.002348 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:54:04.002348 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:54:04.002348 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:54:04.002348 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:54:04.002348 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:54:04.002348 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:54:04.002348 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:54:04.002348 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:54:04.002348 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:54:04.002348 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 13 15:54:04.002348 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 13 15:54:04.002348 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 13 15:54:04.003318 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 13 15:54:04.003318 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:54:04.003318 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:54:04.003318 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:54:04.003318 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:54:04.003318 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:54:04.003318 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:54:04.003318 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:54:04.003318 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application with Lock( Sep 13 15:54:04.003318 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:54:04.003318 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:54:04.003318 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:54:04.003318 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:54:04.003318 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:54:04.003318 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:54:04.003318 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:54:04.003318 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:54:04.004337 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 13 15:54:04.004337 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 13 15:54:04.004337 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 13 15:54:04.004337 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 13 15:54:04.004337 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 13 15:54:04.004337 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 13 15:54:04.004337 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 13 15:54:04.004337 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application Sep 13 15:54:04.008727 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 265/522] 10.222.0.50 () {56 vars in 1111 bytes} [Tue Sep 13 15:54:03 2022] GET /identity/v3/projects/service => generated 88 bytes in 163 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:54:04.009754 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db7d388c-47aa-4acd-8035-f111ba30483d None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:04.053519 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:54:04.110397 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9b07581-3967-4ab5-86cf-767d101fe78e None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:04.123174 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:54:04.142615 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db7d388c-47aa-4acd-8035-f111ba30483d None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:04.151134 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6938c02b-21d3-4d55-969d-2026e14b2efe None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:04.151919 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6938c02b-21d3-4d55-969d-2026e14b2efe None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:04.152206 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6938c02b-21d3-4d55-969d-2026e14b2efe None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:04.152487 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6938c02b-21d3-4d55-969d-2026e14b2efe None admin] PATH_INFO: `/v3/domains` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:04.153146 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6938c02b-21d3-4d55-969d-2026e14b2efe None admin] RBAC: Adding query filter params (name=Default) {{(pid=81438) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:54:04.153521 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6938c02b-21d3-4d55-969d-2026e14b2efe None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:04.155006 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6938c02b-21d3-4d55-969d-2026e14b2efe None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:04.170664 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 258/523] 10.222.0.50 () {56 vars in 1109 bytes} [Tue Sep 13 15:54:03 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:54:04.170664 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6938c02b-21d3-4d55-969d-2026e14b2efe None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:04.187077 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe74461e-ae28-4ea0-8bef-aaea5802c17e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:04.187695 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe74461e-ae28-4ea0-8bef-aaea5802c17e None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:04.187955 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe74461e-ae28-4ea0-8bef-aaea5802c17e None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:04.188220 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe74461e-ae28-4ea0-8bef-aaea5802c17e None admin] PATH_INFO: `/v3/domains` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:04.188858 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe74461e-ae28-4ea0-8bef-aaea5802c17e None admin] RBAC: Adding query filter params (name=Default) {{(pid=81437) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:54:04.189157 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe74461e-ae28-4ea0-8bef-aaea5802c17e None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:04.190527 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe74461e-ae28-4ea0-8bef-aaea5802c17e None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:04.204321 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 266/524] 10.222.0.50 () {56 vars in 1109 bytes} [Tue Sep 13 15:54:04 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 195 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:54:04.205196 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe74461e-ae28-4ea0-8bef-aaea5802c17e None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:04.278458 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6938c02b-21d3-4d55-969d-2026e14b2efe None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:04.304897 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:54:04.315780 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe74461e-ae28-4ea0-8bef-aaea5802c17e None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:04.317906 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f880967a-480d-4fa7-971b-ad2da2cd9578 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:04.318516 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f880967a-480d-4fa7-971b-ad2da2cd9578 None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:04.318790 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f880967a-480d-4fa7-971b-ad2da2cd9578 None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:04.319089 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f880967a-480d-4fa7-971b-ad2da2cd9578 None admin] PATH_INFO: `/v3/domains` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:04.319722 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f880967a-480d-4fa7-971b-ad2da2cd9578 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81438) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:54:04.320020 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f880967a-480d-4fa7-971b-ad2da2cd9578 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:04.321374 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f880967a-480d-4fa7-971b-ad2da2cd9578 None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:04.334434 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 259/525] 10.222.0.50 () {56 vars in 1109 bytes} [Tue Sep 13 15:54:04 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:54:04.340170 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f880967a-480d-4fa7-971b-ad2da2cd9578 None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:04.353133 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11e1192a-fee5-4f79-8774-d6d79546c324 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:04.354715 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11e1192a-fee5-4f79-8774-d6d79546c324 None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:04.355050 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11e1192a-fee5-4f79-8774-d6d79546c324 None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:04.355296 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11e1192a-fee5-4f79-8774-d6d79546c324 None admin] PATH_INFO: `/v3/projects` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:04.355995 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11e1192a-fee5-4f79-8774-d6d79546c324 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81437) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:54:04.356304 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11e1192a-fee5-4f79-8774-d6d79546c324 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:04.357772 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11e1192a-fee5-4f79-8774-d6d79546c324 None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:04.374153 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:54:04.378539 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 267/526] 10.222.0.50 () {56 vars in 1148 bytes} [Tue Sep 13 15:54:04 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:54:04.379712 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11e1192a-fee5-4f79-8774-d6d79546c324 None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:04.456281 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f880967a-480d-4fa7-971b-ad2da2cd9578 None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:04.490645 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11e1192a-fee5-4f79-8774-d6d79546c324 None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:04.497509 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1f031ac-9bbd-45fc-b710-62b8441d8449 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:04.498209 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1f031ac-9bbd-45fc-b710-62b8441d8449 None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:04.498486 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1f031ac-9bbd-45fc-b710-62b8441d8449 None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:04.499256 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1f031ac-9bbd-45fc-b710-62b8441d8449 None admin] PATH_INFO: `/v3/users/neutron` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:04.507954 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1f031ac-9bbd-45fc-b710-62b8441d8449 None admin] RBAC: Authorizing `identity:get_user(user_id=neutron)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:04.509750 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1f031ac-9bbd-45fc-b710-62b8441d8449 None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:04.522474 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application [None req-f1f031ac-9bbd-45fc-b710-62b8441d8449 None admin] Could not find user: neutron.: keystone.exception.UserNotFound: Could not find user: neutron. Sep 13 15:54:04.522474 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:54:04.522474 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:54:04.522474 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:54:04.522474 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:54:04.522474 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:54:04.522474 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:54:04.522474 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:54:04.522474 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:54:04.522474 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:54:04.522474 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:54:04.522474 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:54:04.522474 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 13 15:54:04.522474 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 13 15:54:04.522474 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 13 15:54:04.522474 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 13 15:54:04.523491 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:54:04.523491 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:54:04.523491 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 13 15:54:04.523491 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 13 15:54:04.523491 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 13 15:54:04.523491 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 13 15:54:04.523491 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:54:04.523491 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:54:04.523491 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:54:04.523491 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:54:04.523491 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:54:04.523491 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application with Lock( Sep 13 15:54:04.523491 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:54:04.523491 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:54:04.523491 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:54:04.523491 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:54:04.523491 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:54:04.523491 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:54:04.524465 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:54:04.524465 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:54:04.524465 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 13 15:54:04.524465 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 13 15:54:04.524465 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 13 15:54:04.524465 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 13 15:54:04.524465 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 13 15:54:04.524465 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 13 15:54:04.524465 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: neutron. Sep 13 15:54:04.524465 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Sep 13 15:54:04.524465 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 260/527] 10.222.0.50 () {56 vars in 1102 bytes} [Tue Sep 13 15:54:04 2022] GET /identity/v3/users/neutron => generated 85 bytes in 189 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:54:04.524465 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1f031ac-9bbd-45fc-b710-62b8441d8449 None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:04.531333 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e53074c-eef8-4290-b9d7-1ea5bf460a36 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:04.532058 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e53074c-eef8-4290-b9d7-1ea5bf460a36 None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:04.532337 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e53074c-eef8-4290-b9d7-1ea5bf460a36 None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:04.532606 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e53074c-eef8-4290-b9d7-1ea5bf460a36 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:04.540320 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e53074c-eef8-4290-b9d7-1ea5bf460a36 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:04.542307 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e53074c-eef8-4290-b9d7-1ea5bf460a36 None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:04.553989 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application [None req-8e53074c-eef8-4290-b9d7-1ea5bf460a36 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 13 15:54:04.553989 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:54:04.553989 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:54:04.553989 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:54:04.553989 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:54:04.553989 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:54:04.553989 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:54:04.553989 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:54:04.553989 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:54:04.553989 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:54:04.553989 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:54:04.553989 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:54:04.553989 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 13 15:54:04.553989 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 13 15:54:04.553989 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 13 15:54:04.554837 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 13 15:54:04.554837 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:54:04.554837 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:54:04.554837 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:54:04.554837 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:54:04.554837 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:54:04.554837 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:54:04.554837 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:54:04.554837 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application with Lock( Sep 13 15:54:04.554837 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:54:04.554837 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:54:04.554837 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:54:04.554837 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:54:04.554837 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:54:04.554837 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:54:04.554837 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:54:04.554837 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:54:04.555723 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 13 15:54:04.555723 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 13 15:54:04.555723 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 13 15:54:04.555723 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 13 15:54:04.555723 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 13 15:54:04.555723 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 13 15:54:04.555723 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 13 15:54:04.555723 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application Sep 13 15:54:04.555723 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 268/528] 10.222.0.50 () {56 vars in 1111 bytes} [Tue Sep 13 15:54:04 2022] GET /identity/v3/projects/service => generated 88 bytes in 176 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:54:04.556191 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e53074c-eef8-4290-b9d7-1ea5bf460a36 None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:04.573726 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:54:04.627404 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:54:04.638371 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1f031ac-9bbd-45fc-b710-62b8441d8449 None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:04.661627 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e53074c-eef8-4290-b9d7-1ea5bf460a36 None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:04.673693 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57729a16-bc49-4c75-b957-0a86c46671e6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:04.674325 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57729a16-bc49-4c75-b957-0a86c46671e6 None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:04.674605 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57729a16-bc49-4c75-b957-0a86c46671e6 None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:04.674878 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57729a16-bc49-4c75-b957-0a86c46671e6 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:04.683202 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57729a16-bc49-4c75-b957-0a86c46671e6 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:04.684691 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57729a16-bc49-4c75-b957-0a86c46671e6 None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:04.698160 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c16d1a5b-e93c-4660-a603-a09d29629d76 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:04.698773 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c16d1a5b-e93c-4660-a603-a09d29629d76 None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:04.698998 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application [None req-57729a16-bc49-4c75-b957-0a86c46671e6 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 13 15:54:04.698998 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:54:04.698998 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:54:04.698998 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:54:04.698998 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:54:04.698998 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:54:04.698998 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:54:04.698998 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:54:04.698998 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:54:04.698998 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:54:04.698998 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:54:04.698998 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:54:04.698998 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 13 15:54:04.698998 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 13 15:54:04.698998 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 13 15:54:04.699789 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 13 15:54:04.699789 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:54:04.699789 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:54:04.699789 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:54:04.699789 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:54:04.699789 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:54:04.699789 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:54:04.699789 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:54:04.699789 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application with Lock( Sep 13 15:54:04.699789 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:54:04.699789 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:54:04.699789 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:54:04.699789 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:54:04.699789 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:54:04.699789 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:54:04.699789 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:54:04.699789 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:54:04.700629 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 13 15:54:04.700629 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 13 15:54:04.700629 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 13 15:54:04.700629 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 13 15:54:04.700629 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 13 15:54:04.700629 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 13 15:54:04.700629 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 13 15:54:04.700629 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Sep 13 15:54:04.701539 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c16d1a5b-e93c-4660-a603-a09d29629d76 None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:04.701539 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c16d1a5b-e93c-4660-a603-a09d29629d76 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:04.701539 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c16d1a5b-e93c-4660-a603-a09d29629d76 None admin] RBAC: Adding query filter params (role.id=da847124d9374ac88b27968451c486ab, scope.project.id=5811e3b83a1e462e9320cad79a3ad791, user.id=4e039b46c5e3445bbb39904b00c5430e) {{(pid=81437) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:54:04.701539 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c16d1a5b-e93c-4660-a603-a09d29629d76 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:04.701890 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c16d1a5b-e93c-4660-a603-a09d29629d76 None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:04.705135 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 261/529] 10.222.0.50 () {56 vars in 1111 bytes} [Tue Sep 13 15:54:04 2022] GET /identity/v3/projects/service => generated 88 bytes in 180 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:54:04.705602 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57729a16-bc49-4c75-b957-0a86c46671e6 None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:04.719219 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 269/530] 10.222.0.50 () {56 vars in 1374 bytes} [Tue Sep 13 15:54:04 2022] GET /identity/v3/role_assignments?user.id=4e039b46c5e3445bbb39904b00c5430e&scope.project.id=5811e3b83a1e462e9320cad79a3ad791&role.id=da847124d9374ac88b27968451c486ab => generated 259 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:54:04.719219 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c16d1a5b-e93c-4660-a603-a09d29629d76 None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:04.815358 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:54:04.843985 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57729a16-bc49-4c75-b957-0a86c46671e6 None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:04.855238 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c16d1a5b-e93c-4660-a603-a09d29629d76 None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:04.885081 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22ac76fd-8556-40f1-83f6-5a4a9c505007 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:04.885524 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22ac76fd-8556-40f1-83f6-5a4a9c505007 None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:04.885870 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22ac76fd-8556-40f1-83f6-5a4a9c505007 None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:04.886152 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22ac76fd-8556-40f1-83f6-5a4a9c505007 None admin] PATH_INFO: `/v3/users` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:04.886997 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22ac76fd-8556-40f1-83f6-5a4a9c505007 None admin] RBAC: Adding query filter params (domain_id=default, name=neutron) {{(pid=81438) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:54:04.887409 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22ac76fd-8556-40f1-83f6-5a4a9c505007 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:04.888781 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22ac76fd-8556-40f1-83f6-5a4a9c505007 None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:04.904579 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd39a884-4bcc-412d-839f-cc48b000da95 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:04.905991 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd39a884-4bcc-412d-839f-cc48b000da95 None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:04.906277 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd39a884-4bcc-412d-839f-cc48b000da95 None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:04.906554 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd39a884-4bcc-412d-839f-cc48b000da95 None admin] PATH_INFO: `/v3/projects` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:04.907449 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd39a884-4bcc-412d-839f-cc48b000da95 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81437) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:54:04.908398 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd39a884-4bcc-412d-839f-cc48b000da95 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:04.909254 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd39a884-4bcc-412d-839f-cc48b000da95 None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:04.909831 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 262/531] 10.222.0.50 () {56 vars in 1139 bytes} [Tue Sep 13 15:54:04 2022] GET /identity/v3/users?domain_id=default&name=neutron => generated 374 bytes in 204 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:54:04.914847 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22ac76fd-8556-40f1-83f6-5a4a9c505007 None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:04.925140 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 270/532] 10.222.0.50 () {56 vars in 1148 bytes} [Tue Sep 13 15:54:04 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 216 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:54:04.925856 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:54:04.926038 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd39a884-4bcc-412d-839f-cc48b000da95 None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:05.041427 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd39a884-4bcc-412d-839f-cc48b000da95 None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:05.043414 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22ac76fd-8556-40f1-83f6-5a4a9c505007 None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:05.090181 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06c2ad5a-9db6-4935-9dba-0ebc904d57d2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:05.091105 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06c2ad5a-9db6-4935-9dba-0ebc904d57d2 None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:05.091591 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06c2ad5a-9db6-4935-9dba-0ebc904d57d2 None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:05.092206 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06c2ad5a-9db6-4935-9dba-0ebc904d57d2 None admin] PATH_INFO: `/v3/projects` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:05.093072 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06c2ad5a-9db6-4935-9dba-0ebc904d57d2 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81438) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:54:05.093407 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24ea5fff-24f6-4a8a-86d5-e26bfe0c7711 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:05.093771 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06c2ad5a-9db6-4935-9dba-0ebc904d57d2 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:05.094157 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24ea5fff-24f6-4a8a-86d5-e26bfe0c7711 None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:05.094497 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24ea5fff-24f6-4a8a-86d5-e26bfe0c7711 None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:05.094833 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24ea5fff-24f6-4a8a-86d5-e26bfe0c7711 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:05.096052 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06c2ad5a-9db6-4935-9dba-0ebc904d57d2 None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:05.098764 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:54:05.124978 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 263/533] 10.222.0.50 () {56 vars in 1148 bytes} [Tue Sep 13 15:54:04 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 210 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:54:05.126736 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06c2ad5a-9db6-4935-9dba-0ebc904d57d2 None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:05.180585 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24ea5fff-24f6-4a8a-86d5-e26bfe0c7711 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:05.183168 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24ea5fff-24f6-4a8a-86d5-e26bfe0c7711 None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:05.209325 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:54:05.249979 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06c2ad5a-9db6-4935-9dba-0ebc904d57d2 None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:05.296282 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d2bde57-e94c-45e1-aa3b-f9ff8ee5a7ee None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:05.296896 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d2bde57-e94c-45e1-aa3b-f9ff8ee5a7ee None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:05.297180 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d2bde57-e94c-45e1-aa3b-f9ff8ee5a7ee None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:05.297455 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d2bde57-e94c-45e1-aa3b-f9ff8ee5a7ee None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:05.297875 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 271/534] 10.222.0.50 () {58 vars in 1282 bytes} [Tue Sep 13 15:54:04 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 372 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 13 15:54:05.298319 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d2bde57-e94c-45e1-aa3b-f9ff8ee5a7ee None admin] RBAC: Adding query filter params (role.id=da847124d9374ac88b27968451c486ab, scope.project.id=5811e3b83a1e462e9320cad79a3ad791, user.id=7fd6b5afd67b48f9b0f0597ccd4cf4df) {{(pid=81438) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:54:05.298592 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d2bde57-e94c-45e1-aa3b-f9ff8ee5a7ee None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:05.298873 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24ea5fff-24f6-4a8a-86d5-e26bfe0c7711 None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:05.299966 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d2bde57-e94c-45e1-aa3b-f9ff8ee5a7ee None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:05.307502 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 264/535] 10.222.0.50 () {56 vars in 1374 bytes} [Tue Sep 13 15:54:05 2022] GET /identity/v3/role_assignments?user.id=7fd6b5afd67b48f9b0f0597ccd4cf4df&scope.project.id=5811e3b83a1e462e9320cad79a3ad791&role.id=da847124d9374ac88b27968451c486ab => generated 259 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:54:05.308646 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d2bde57-e94c-45e1-aa3b-f9ff8ee5a7ee None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:05.377156 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:54:05.431585 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d2bde57-e94c-45e1-aa3b-f9ff8ee5a7ee None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:05.472429 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:54:05.487208 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfec1ddd-0a14-4216-8708-c0b6a0335e8a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:05.488194 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfec1ddd-0a14-4216-8708-c0b6a0335e8a None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:05.488653 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfec1ddd-0a14-4216-8708-c0b6a0335e8a None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:05.489089 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfec1ddd-0a14-4216-8708-c0b6a0335e8a None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:05.512689 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cfec1ddd-0a14-4216-8708-c0b6a0335e8a None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:05.514179 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cfec1ddd-0a14-4216-8708-c0b6a0335e8a None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:05.525068 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24ea5fff-24f6-4a8a-86d5-e26bfe0c7711 None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:05.552169 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application [None req-cfec1ddd-0a14-4216-8708-c0b6a0335e8a None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:54:05.552169 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:54:05.552169 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:54:05.552169 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:54:05.552169 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:54:05.552169 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:54:05.552169 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:54:05.552169 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:54:05.552169 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:54:05.552169 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:54:05.552169 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:54:05.552169 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:54:05.552169 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 13 15:54:05.552169 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 13 15:54:05.552169 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 13 15:54:05.552169 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 13 15:54:05.553127 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:54:05.553127 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:54:05.553127 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:54:05.553127 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:54:05.553127 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:54:05.553127 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:54:05.553127 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:54:05.553127 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application with Lock( Sep 13 15:54:05.553127 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:54:05.553127 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:54:05.553127 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:54:05.553127 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:54:05.553127 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:54:05.553127 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:54:05.553127 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:54:05.553127 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:54:05.553127 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 13 15:54:05.554563 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 13 15:54:05.554563 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:54:05.554563 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Sep 13 15:54:05.554563 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 265/536] 10.222.0.50 () {56 vars in 1108 bytes} [Tue Sep 13 15:54:05 2022] GET /identity/v3/domains/Default => generated 87 bytes in 245 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:54:05.559533 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfec1ddd-0a14-4216-8708-c0b6a0335e8a None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:05.598040 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-733a0bab-dfd5-4f2b-8648-e06dec817e29 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:05.598653 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-733a0bab-dfd5-4f2b-8648-e06dec817e29 None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:05.598969 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-733a0bab-dfd5-4f2b-8648-e06dec817e29 None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:05.599250 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-733a0bab-dfd5-4f2b-8648-e06dec817e29 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:05.599976 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-733a0bab-dfd5-4f2b-8648-e06dec817e29 None admin] RBAC: Adding query filter params (role.id=da847124d9374ac88b27968451c486ab, scope.project.id=5811e3b83a1e462e9320cad79a3ad791, user.id=6ab5f8018f464be69d4af7683f553738) {{(pid=81437) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:54:05.600284 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-733a0bab-dfd5-4f2b-8648-e06dec817e29 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:05.601859 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-733a0bab-dfd5-4f2b-8648-e06dec817e29 None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:05.614858 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 272/537] 10.222.0.50 () {56 vars in 1374 bytes} [Tue Sep 13 15:54:05 2022] GET /identity/v3/role_assignments?user.id=6ab5f8018f464be69d4af7683f553738&scope.project.id=5811e3b83a1e462e9320cad79a3ad791&role.id=da847124d9374ac88b27968451c486ab => generated 259 bytes in 315 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:54:05.685074 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfec1ddd-0a14-4216-8708-c0b6a0335e8a None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:05.738759 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-115567cc-7470-42b1-a7c9-42dd2c13c1a2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:05.739425 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-115567cc-7470-42b1-a7c9-42dd2c13c1a2 None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:05.739713 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-115567cc-7470-42b1-a7c9-42dd2c13c1a2 None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:05.739987 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-115567cc-7470-42b1-a7c9-42dd2c13c1a2 None admin] PATH_INFO: `/v3/domains` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:05.740651 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-115567cc-7470-42b1-a7c9-42dd2c13c1a2 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81438) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:54:05.740958 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-115567cc-7470-42b1-a7c9-42dd2c13c1a2 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:05.742392 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-115567cc-7470-42b1-a7c9-42dd2c13c1a2 None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:05.756962 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 266/538] 10.222.0.50 () {56 vars in 1109 bytes} [Tue Sep 13 15:54:05 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 198 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 13 15:54:05.762739 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-733a0bab-dfd5-4f2b-8648-e06dec817e29 None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:05.935814 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:54:05.938342 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-733a0bab-dfd5-4f2b-8648-e06dec817e29 None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:05.981894 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8ea381f-cb11-444f-a9b8-7b0b55f2ec15 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:05.982531 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8ea381f-cb11-444f-a9b8-7b0b55f2ec15 None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:05.982815 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8ea381f-cb11-444f-a9b8-7b0b55f2ec15 None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:05.983116 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8ea381f-cb11-444f-a9b8-7b0b55f2ec15 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:05.990120 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8ea381f-cb11-444f-a9b8-7b0b55f2ec15 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:05.991675 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8ea381f-cb11-444f-a9b8-7b0b55f2ec15 None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:06.004688 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application [None req-f8ea381f-cb11-444f-a9b8-7b0b55f2ec15 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 13 15:54:06.004688 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:54:06.004688 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:54:06.004688 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:54:06.004688 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:54:06.004688 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:54:06.004688 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:54:06.004688 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:54:06.004688 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:54:06.004688 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:54:06.004688 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:54:06.004688 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:54:06.004688 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 13 15:54:06.004688 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 13 15:54:06.004688 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 13 15:54:06.005569 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 13 15:54:06.005569 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:54:06.005569 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:54:06.005569 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:54:06.005569 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:54:06.005569 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:54:06.005569 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:54:06.005569 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:54:06.005569 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application with Lock( Sep 13 15:54:06.005569 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:54:06.005569 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:54:06.005569 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:54:06.005569 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:54:06.005569 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:54:06.005569 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:54:06.005569 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:54:06.005569 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:54:06.006635 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 13 15:54:06.006635 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 13 15:54:06.006635 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 13 15:54:06.006635 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 13 15:54:06.006635 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 13 15:54:06.006635 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 13 15:54:06.006635 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 13 15:54:06.006635 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application Sep 13 15:54:06.006635 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 273/539] 10.222.0.50 () {56 vars in 1111 bytes} [Tue Sep 13 15:54:05 2022] GET /identity/v3/projects/service => generated 88 bytes in 244 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:54:06.016040 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-115567cc-7470-42b1-a7c9-42dd2c13c1a2 None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:06.097575 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:54:06.171174 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-115567cc-7470-42b1-a7c9-42dd2c13c1a2 None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:06.219426 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b76d6ae8-5fdb-4889-ba3a-15a1c7721878 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:06.220045 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b76d6ae8-5fdb-4889-ba3a-15a1c7721878 None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:06.220246 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b76d6ae8-5fdb-4889-ba3a-15a1c7721878 None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:06.220494 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b76d6ae8-5fdb-4889-ba3a-15a1c7721878 None admin] PATH_INFO: `/v3/projects` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:06.221184 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b76d6ae8-5fdb-4889-ba3a-15a1c7721878 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81438) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:54:06.221491 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b76d6ae8-5fdb-4889-ba3a-15a1c7721878 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:06.222925 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b76d6ae8-5fdb-4889-ba3a-15a1c7721878 None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:06.237845 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 267/540] 10.222.0.50 () {56 vars in 1148 bytes} [Tue Sep 13 15:54:06 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 222 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:54:06.243495 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8ea381f-cb11-444f-a9b8-7b0b55f2ec15 None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:06.412721 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ff56383-72e9-4def-b08c-d6b9fa9dbf44 None None] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:06.413045 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ff56383-72e9-4def-b08c-d6b9fa9dbf44 None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:06.413289 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ff56383-72e9-4def-b08c-d6b9fa9dbf44 None None] PATH_INFO: `/` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:06.414886 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 268/541] 10.222.0.50 () {54 vars in 927 bytes} [Tue Sep 13 15:54:06 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:54:06.426480 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaf1b9f5-c440-4525-8ca5-1d08efda4ef2 None None] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:06.426825 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaf1b9f5-c440-4525-8ca5-1d08efda4ef2 None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:06.427311 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaf1b9f5-c440-4525-8ca5-1d08efda4ef2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:06.430104 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8ea381f-cb11-444f-a9b8-7b0b55f2ec15 None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:06.468572 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e94cabd-0611-45cf-9764-4ee161a3b30c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:06.469184 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e94cabd-0611-45cf-9764-4ee161a3b30c None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:06.469513 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e94cabd-0611-45cf-9764-4ee161a3b30c None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:06.469738 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e94cabd-0611-45cf-9764-4ee161a3b30c None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:06.470459 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e94cabd-0611-45cf-9764-4ee161a3b30c None admin] RBAC: Adding query filter params (role.id=da847124d9374ac88b27968451c486ab, scope.project.id=5811e3b83a1e462e9320cad79a3ad791, user.id=2f436f07939d4855a21b195bc7566244) {{(pid=81437) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:54:06.470773 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e94cabd-0611-45cf-9764-4ee161a3b30c None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:06.471906 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.auth.core [None req-eaf1b9f5-c440-4525-8ca5-1d08efda4ef2 None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81438) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:54:06.472248 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e94cabd-0611-45cf-9764-4ee161a3b30c None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:06.473127 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-eaf1b9f5-c440-4525-8ca5-1d08efda4ef2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:54:06.478385 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 274/542] 10.222.0.50 () {56 vars in 1374 bytes} [Tue Sep 13 15:54:06 2022] GET /identity/v3/role_assignments?user.id=2f436f07939d4855a21b195bc7566244&scope.project.id=5811e3b83a1e462e9320cad79a3ad791&role.id=da847124d9374ac88b27968451c486ab => generated 259 bytes in 235 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:54:06.593792 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 269/543] 10.222.0.50 () {58 vars in 1024 bytes} [Tue Sep 13 15:54:06 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 168 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 13 15:54:06.980441 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e94cabd-0611-45cf-9764-4ee161a3b30c None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:07.008292 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:54:07.117761 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e94cabd-0611-45cf-9764-4ee161a3b30c None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:07.163841 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1061e323-0cbe-447f-8754-0529d9a7fe34 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:07.164477 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1061e323-0cbe-447f-8754-0529d9a7fe34 None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:07.164761 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1061e323-0cbe-447f-8754-0529d9a7fe34 None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:07.165040 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1061e323-0cbe-447f-8754-0529d9a7fe34 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:07.185831 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1061e323-0cbe-447f-8754-0529d9a7fe34 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 13 15:54:07.191139 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application [None req-1061e323-0cbe-447f-8754-0529d9a7fe34 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 13 15:54:07.191139 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:54:07.191139 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:54:07.191139 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:54:07.191139 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:54:07.191139 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:54:07.191139 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:54:07.191139 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:54:07.191139 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:54:07.191139 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:54:07.191139 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:54:07.191139 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:54:07.191139 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 13 15:54:07.191139 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 13 15:54:07.191139 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 13 15:54:07.191139 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 13 15:54:07.192022 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 13 15:54:07.192022 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 13 15:54:07.192022 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 13 15:54:07.192022 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 13 15:54:07.192022 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:54:07.192022 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:54:07.192022 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:54:07.192022 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:54:07.192022 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:54:07.192022 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:54:07.192022 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:54:07.192022 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application with Lock( Sep 13 15:54:07.192022 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:54:07.192022 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:54:07.192022 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:54:07.192022 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:54:07.192805 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:54:07.192805 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:54:07.192805 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:54:07.192805 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:54:07.192805 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 13 15:54:07.192805 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 13 15:54:07.192805 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 13 15:54:07.192805 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 13 15:54:07.192805 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 13 15:54:07.192805 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 13 15:54:07.192805 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 13 15:54:07.192805 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application Sep 13 15:54:07.192805 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 275/544] 10.222.0.50 () {56 vars in 1102 bytes} [Tue Sep 13 15:54:06 2022] GET /identity/v3/roles/service => generated 85 bytes in 213 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:54:07.195096 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-243599e7-9942-4596-9d2b-ec3bac79596d None None] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:07.195387 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-243599e7-9942-4596-9d2b-ec3bac79596d None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:07.195662 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-243599e7-9942-4596-9d2b-ec3bac79596d None None] PATH_INFO: `/` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:07.197914 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 270/545] 10.222.0.50 () {54 vars in 927 bytes} [Tue Sep 13 15:54:07 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:54:07.198763 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1061e323-0cbe-447f-8754-0529d9a7fe34 None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:07.202959 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-579a97fe-cb0a-411c-bdf0-7e8df6fee631 None None] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:07.203252 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-579a97fe-cb0a-411c-bdf0-7e8df6fee631 None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:07.203532 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-579a97fe-cb0a-411c-bdf0-7e8df6fee631 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:07.227889 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.auth.core [None req-579a97fe-cb0a-411c-bdf0-7e8df6fee631 None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81438) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:54:07.229120 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-579a97fe-cb0a-411c-bdf0-7e8df6fee631 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:54:07.304381 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:54:07.320842 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:54:07.333528 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1061e323-0cbe-447f-8754-0529d9a7fe34 None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:07.373078 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa59ca6d-056e-4618-8f19-7c03c0ede09b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:07.373841 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa59ca6d-056e-4618-8f19-7c03c0ede09b None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:07.374222 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa59ca6d-056e-4618-8f19-7c03c0ede09b None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:07.374601 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa59ca6d-056e-4618-8f19-7c03c0ede09b None admin] PATH_INFO: `/v3/roles` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:07.375555 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa59ca6d-056e-4618-8f19-7c03c0ede09b None admin] RBAC: Adding query filter params (name=service) {{(pid=81437) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:54:07.375966 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa59ca6d-056e-4618-8f19-7c03c0ede09b None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:07.377795 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa59ca6d-056e-4618-8f19-7c03c0ede09b None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:07.388506 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 276/546] 10.222.0.50 () {56 vars in 1103 bytes} [Tue Sep 13 15:54:07 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 190 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:54:07.396004 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa59ca6d-056e-4618-8f19-7c03c0ede09b None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:07.416360 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 271/547] 10.222.0.50 () {58 vars in 1024 bytes} [Tue Sep 13 15:54:07 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 214 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 13 15:54:07.551598 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa59ca6d-056e-4618-8f19-7c03c0ede09b None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:07.602768 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e54b57bc-dd9c-4148-bf4c-2f2707f04858 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:07.603468 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e54b57bc-dd9c-4148-bf4c-2f2707f04858 None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:07.603750 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e54b57bc-dd9c-4148-bf4c-2f2707f04858 None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:07.604030 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e54b57bc-dd9c-4148-bf4c-2f2707f04858 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:07.620139 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e54b57bc-dd9c-4148-bf4c-2f2707f04858 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:07.621599 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e54b57bc-dd9c-4148-bf4c-2f2707f04858 None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:07.629523 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:54:07.644210 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application [None req-e54b57bc-dd9c-4148-bf4c-2f2707f04858 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:54:07.644210 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:54:07.644210 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:54:07.644210 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:54:07.644210 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:54:07.644210 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:54:07.644210 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:54:07.644210 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:54:07.644210 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:54:07.644210 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:54:07.644210 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:54:07.644210 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:54:07.644210 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 13 15:54:07.644210 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 13 15:54:07.644210 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 13 15:54:07.644210 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 13 15:54:07.645140 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:54:07.645140 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:54:07.645140 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:54:07.645140 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:54:07.645140 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:54:07.645140 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:54:07.645140 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:54:07.645140 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application with Lock( Sep 13 15:54:07.645140 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:54:07.645140 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:54:07.645140 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:54:07.645140 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:54:07.645140 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:54:07.645140 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:54:07.645140 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:54:07.645140 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:54:07.645140 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 13 15:54:07.645986 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 13 15:54:07.645986 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:54:07.645986 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application Sep 13 15:54:07.645986 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 277/548] 10.222.0.50 () {56 vars in 1108 bytes} [Tue Sep 13 15:54:07 2022] GET /identity/v3/domains/Default => generated 87 bytes in 250 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:54:07.651789 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-579a97fe-cb0a-411c-bdf0-7e8df6fee631 None None] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:07.692026 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e54b57bc-dd9c-4148-bf4c-2f2707f04858 None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:07.813937 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-579a97fe-cb0a-411c-bdf0-7e8df6fee631 None None] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:07.880213 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ba273ea-57aa-4008-a262-371dc28de5e4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:07.880743 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ba273ea-57aa-4008-a262-371dc28de5e4 None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:07.881025 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ba273ea-57aa-4008-a262-371dc28de5e4 None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:07.881300 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ba273ea-57aa-4008-a262-371dc28de5e4 None admin] PATH_INFO: `/v3/domains` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:07.881946 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ba273ea-57aa-4008-a262-371dc28de5e4 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81438) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:54:07.882248 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ba273ea-57aa-4008-a262-371dc28de5e4 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:07.883670 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ba273ea-57aa-4008-a262-371dc28de5e4 None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:07.899104 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e54b57bc-dd9c-4148-bf4c-2f2707f04858 None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:07.907533 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 272/549] 10.222.0.50 () {56 vars in 1109 bytes} [Tue Sep 13 15:54:07 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 256 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:54:07.913060 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ba273ea-57aa-4008-a262-371dc28de5e4 None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:07.941842 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f5d56c6-e5ff-4564-b477-525b2a275ec7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:07.942506 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f5d56c6-e5ff-4564-b477-525b2a275ec7 None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:07.942789 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f5d56c6-e5ff-4564-b477-525b2a275ec7 None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:07.943100 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f5d56c6-e5ff-4564-b477-525b2a275ec7 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:07.957935 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f5d56c6-e5ff-4564-b477-525b2a275ec7 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 13 15:54:07.964708 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application [None req-6f5d56c6-e5ff-4564-b477-525b2a275ec7 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 13 15:54:07.964708 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:54:07.964708 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:54:07.964708 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:54:07.964708 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:54:07.964708 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:54:07.964708 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:54:07.964708 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:54:07.964708 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:54:07.964708 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:54:07.964708 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:54:07.964708 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:54:07.964708 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 13 15:54:07.964708 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 13 15:54:07.964708 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 13 15:54:07.964708 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 13 15:54:07.966206 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 13 15:54:07.966206 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 13 15:54:07.966206 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 13 15:54:07.966206 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 13 15:54:07.966206 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:54:07.966206 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:54:07.966206 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:54:07.966206 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:54:07.966206 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:54:07.966206 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:54:07.966206 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:54:07.966206 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application with Lock( Sep 13 15:54:07.966206 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:54:07.966206 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:54:07.966206 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:54:07.966206 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:54:07.967595 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:54:07.967595 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:54:07.967595 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:54:07.967595 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:54:07.967595 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 13 15:54:07.967595 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 13 15:54:07.967595 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 13 15:54:07.967595 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 13 15:54:07.967595 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 13 15:54:07.967595 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 13 15:54:07.967595 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 13 15:54:07.967595 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application Sep 13 15:54:07.970883 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 278/550] 10.222.0.50 () {56 vars in 1102 bytes} [Tue Sep 13 15:54:07 2022] GET /identity/v3/roles/service => generated 85 bytes in 275 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:54:07.972859 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f5d56c6-e5ff-4564-b477-525b2a275ec7 None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:07.976645 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:54:08.095057 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:54:08.122268 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ba273ea-57aa-4008-a262-371dc28de5e4 None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:08.148745 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f5d56c6-e5ff-4564-b477-525b2a275ec7 None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:08.161765 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c92973b7-9c93-4445-9c69-8307ef820c04 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:08.163009 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c92973b7-9c93-4445-9c69-8307ef820c04 None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:08.164019 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c92973b7-9c93-4445-9c69-8307ef820c04 None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:08.164655 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c92973b7-9c93-4445-9c69-8307ef820c04 None admin] PATH_INFO: `/v3/users/swift` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:08.174047 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c92973b7-9c93-4445-9c69-8307ef820c04 None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:08.176455 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c92973b7-9c93-4445-9c69-8307ef820c04 None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:08.186133 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2fc1ef3-dd7c-45fb-b4bd-ede2f855dc0d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:08.186875 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2fc1ef3-dd7c-45fb-b4bd-ede2f855dc0d None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:08.187264 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2fc1ef3-dd7c-45fb-b4bd-ede2f855dc0d None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:08.187615 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2fc1ef3-dd7c-45fb-b4bd-ede2f855dc0d None admin] PATH_INFO: `/v3/roles` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:08.188344 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2fc1ef3-dd7c-45fb-b4bd-ede2f855dc0d None admin] RBAC: Adding query filter params (name=service) {{(pid=81437) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:54:08.188719 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2fc1ef3-dd7c-45fb-b4bd-ede2f855dc0d None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:08.190179 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2fc1ef3-dd7c-45fb-b4bd-ede2f855dc0d None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:08.191205 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application [None req-c92973b7-9c93-4445-9c69-8307ef820c04 None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Sep 13 15:54:08.191205 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:54:08.191205 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:54:08.191205 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:54:08.191205 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:54:08.191205 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:54:08.191205 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:54:08.191205 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:54:08.191205 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:54:08.191205 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:54:08.191205 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:54:08.191205 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:54:08.191205 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 13 15:54:08.191205 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 13 15:54:08.191205 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 13 15:54:08.191205 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 13 15:54:08.192275 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:54:08.192275 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:54:08.192275 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 13 15:54:08.192275 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 13 15:54:08.192275 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 13 15:54:08.192275 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 13 15:54:08.192275 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:54:08.192275 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:54:08.192275 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:54:08.192275 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:54:08.192275 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:54:08.192275 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application with Lock( Sep 13 15:54:08.192275 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:54:08.192275 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:54:08.192275 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:54:08.192275 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:54:08.192275 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:54:08.192275 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:54:08.193196 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:54:08.193196 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:54:08.193196 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 13 15:54:08.193196 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 13 15:54:08.193196 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 13 15:54:08.193196 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 13 15:54:08.193196 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 13 15:54:08.193196 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 13 15:54:08.193196 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Sep 13 15:54:08.193196 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Sep 13 15:54:08.195447 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 273/551] 10.222.0.50 () {56 vars in 1096 bytes} [Tue Sep 13 15:54:07 2022] GET /identity/v3/users/swift => generated 83 bytes in 283 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:54:08.197503 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7151fd34-faa9-450b-bd72-2376e737f47d None None] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:08.197994 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7151fd34-faa9-450b-bd72-2376e737f47d None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:08.198496 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7151fd34-faa9-450b-bd72-2376e737f47d None None] PATH_INFO: `/` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:08.204001 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 279/553] 10.222.0.50 () {56 vars in 1103 bytes} [Tue Sep 13 15:54:07 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 228 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 13 15:54:08.204001 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2fc1ef3-dd7c-45fb-b4bd-ede2f855dc0d None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:08.204315 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 274/553] 10.222.0.50 () {54 vars in 927 bytes} [Tue Sep 13 15:54:08 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:54:08.207146 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-373a8236-23e0-48dd-b516-0e563971226e None None] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:08.207614 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-373a8236-23e0-48dd-b516-0e563971226e None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:08.208097 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-373a8236-23e0-48dd-b516-0e563971226e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:08.231018 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.auth.core [None req-373a8236-23e0-48dd-b516-0e563971226e None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81438) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:54:08.233210 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-373a8236-23e0-48dd-b516-0e563971226e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:54:08.326930 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 275/554] 10.222.0.50 () {58 vars in 1024 bytes} [Tue Sep 13 15:54:08 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 115 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 13 15:54:08.326930 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-373a8236-23e0-48dd-b516-0e563971226e None None] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:08.342978 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2fc1ef3-dd7c-45fb-b4bd-ede2f855dc0d None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:08.353057 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:54:08.384004 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:54:08.414046 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b130e3cd-cd36-4cdb-8d83-d78621b7574f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:08.414579 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b130e3cd-cd36-4cdb-8d83-d78621b7574f None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:08.414858 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b130e3cd-cd36-4cdb-8d83-d78621b7574f None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:08.415196 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b130e3cd-cd36-4cdb-8d83-d78621b7574f None admin] PATH_INFO: `/v3/users` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:08.416646 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b130e3cd-cd36-4cdb-8d83-d78621b7574f None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=81437) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:54:08.416996 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b130e3cd-cd36-4cdb-8d83-d78621b7574f None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:08.420121 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b130e3cd-cd36-4cdb-8d83-d78621b7574f None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:08.440828 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 280/555] 10.222.0.50 () {56 vars in 1135 bytes} [Tue Sep 13 15:54:08 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 239 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 13 15:54:08.442166 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2f487e3-d7b1-4010-af4c-2fe7aad20862 None None] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:08.442454 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2f487e3-d7b1-4010-af4c-2fe7aad20862 None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:08.442728 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2f487e3-d7b1-4010-af4c-2fe7aad20862 None None] PATH_INFO: `/` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:08.444340 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 281/556] 10.222.0.50 () {54 vars in 927 bytes} [Tue Sep 13 15:54:08 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:54:08.445730 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2f487e3-d7b1-4010-af4c-2fe7aad20862 None None] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:08.485999 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-373a8236-23e0-48dd-b516-0e563971226e None None] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:08.526378 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09cfbc9f-00d8-41b4-acff-2e3e911a6132 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:08.527044 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09cfbc9f-00d8-41b4-acff-2e3e911a6132 None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:08.527490 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09cfbc9f-00d8-41b4-acff-2e3e911a6132 None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:08.527869 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09cfbc9f-00d8-41b4-acff-2e3e911a6132 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:08.538849 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09cfbc9f-00d8-41b4-acff-2e3e911a6132 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:08.540325 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09cfbc9f-00d8-41b4-acff-2e3e911a6132 None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:08.561004 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application [None req-09cfbc9f-00d8-41b4-acff-2e3e911a6132 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:54:08.561004 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:54:08.561004 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:54:08.561004 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:54:08.561004 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:54:08.561004 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:54:08.561004 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:54:08.561004 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:54:08.561004 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:54:08.561004 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:54:08.561004 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:54:08.561004 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:54:08.561004 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 13 15:54:08.561004 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 13 15:54:08.561004 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 13 15:54:08.561004 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 13 15:54:08.562040 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:54:08.562040 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:54:08.562040 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:54:08.562040 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:54:08.562040 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:54:08.562040 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:54:08.562040 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:54:08.562040 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application with Lock( Sep 13 15:54:08.562040 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:54:08.562040 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:54:08.562040 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:54:08.562040 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:54:08.562040 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:54:08.562040 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:54:08.562040 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:54:08.562040 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:54:08.562040 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 13 15:54:08.562940 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 13 15:54:08.562940 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:54:08.562940 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Sep 13 15:54:08.566994 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 276/557] 10.222.0.50 () {56 vars in 1108 bytes} [Tue Sep 13 15:54:08 2022] GET /identity/v3/domains/Default => generated 87 bytes in 243 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:54:08.567728 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2f487e3-d7b1-4010-af4c-2fe7aad20862 None None] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:08.568288 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cea04f3-c31f-4d05-ae94-9421b70ee673 None None] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:08.568511 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cea04f3-c31f-4d05-ae94-9421b70ee673 None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:08.568788 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cea04f3-c31f-4d05-ae94-9421b70ee673 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:08.588256 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.auth.core [None req-8cea04f3-c31f-4d05-ae94-9421b70ee673 None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81438) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:54:08.589560 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-8cea04f3-c31f-4d05-ae94-9421b70ee673 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:54:08.612166 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c2fa141-d2f0-489d-9895-83acc352af43 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:08.612785 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c2fa141-d2f0-489d-9895-83acc352af43 None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:08.613071 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c2fa141-d2f0-489d-9895-83acc352af43 None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:08.613510 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c2fa141-d2f0-489d-9895-83acc352af43 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:08.658343 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:54:08.663996 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:54:08.667647 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 277/558] 10.222.0.50 () {58 vars in 1024 bytes} [Tue Sep 13 15:54:08 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 13 15:54:08.675779 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cea04f3-c31f-4d05-ae94-9421b70ee673 None None] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:08.689927 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c2fa141-d2f0-489d-9895-83acc352af43 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:08.691541 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c2fa141-d2f0-489d-9895-83acc352af43 None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:08.794684 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cea04f3-c31f-4d05-ae94-9421b70ee673 None None] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:08.814070 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 282/559] 10.222.0.50 () {58 vars in 1282 bytes} [Tue Sep 13 15:54:08 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 368 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Sep 13 15:54:08.815954 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c2fa141-d2f0-489d-9895-83acc352af43 None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:08.834104 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64d25010-f0e5-4a9d-a0bb-0a244cbad0bb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:08.834726 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64d25010-f0e5-4a9d-a0bb-0a244cbad0bb None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:08.835053 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64d25010-f0e5-4a9d-a0bb-0a244cbad0bb None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:08.835331 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64d25010-f0e5-4a9d-a0bb-0a244cbad0bb None admin] PATH_INFO: `/v3/domains` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:08.836042 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64d25010-f0e5-4a9d-a0bb-0a244cbad0bb None admin] RBAC: Adding query filter params (name=Default) {{(pid=81438) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:54:08.836353 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64d25010-f0e5-4a9d-a0bb-0a244cbad0bb None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:08.837981 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64d25010-f0e5-4a9d-a0bb-0a244cbad0bb None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:08.852713 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 278/560] 10.222.0.50 () {56 vars in 1109 bytes} [Tue Sep 13 15:54:08 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:54:08.854701 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64d25010-f0e5-4a9d-a0bb-0a244cbad0bb None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:08.918106 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:54:08.930590 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:54:08.943108 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c2fa141-d2f0-489d-9895-83acc352af43 None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:08.982768 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20144948-b591-4047-94b8-ba8228d40ecc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:08.983457 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20144948-b591-4047-94b8-ba8228d40ecc None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:08.983564 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64d25010-f0e5-4a9d-a0bb-0a244cbad0bb None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:08.983729 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20144948-b591-4047-94b8-ba8228d40ecc None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:08.984003 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20144948-b591-4047-94b8-ba8228d40ecc None admin] PATH_INFO: `/v3/roles/service` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:08.996359 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20144948-b591-4047-94b8-ba8228d40ecc None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 13 15:54:09.001198 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application [None req-20144948-b591-4047-94b8-ba8228d40ecc None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 13 15:54:09.001198 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:54:09.001198 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:54:09.001198 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:54:09.001198 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:54:09.001198 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:54:09.001198 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:54:09.001198 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:54:09.001198 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:54:09.001198 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:54:09.001198 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:54:09.001198 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:54:09.001198 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 13 15:54:09.001198 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 13 15:54:09.001198 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 13 15:54:09.001198 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 13 15:54:09.002149 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 13 15:54:09.002149 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 13 15:54:09.002149 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 13 15:54:09.002149 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 13 15:54:09.002149 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:54:09.002149 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:54:09.002149 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:54:09.002149 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:54:09.002149 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:54:09.002149 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:54:09.002149 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:54:09.002149 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application with Lock( Sep 13 15:54:09.002149 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:54:09.002149 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:54:09.002149 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:54:09.002149 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:54:09.003557 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:54:09.003557 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:54:09.003557 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:54:09.003557 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:54:09.003557 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 13 15:54:09.003557 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 13 15:54:09.003557 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 13 15:54:09.003557 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 13 15:54:09.003557 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 13 15:54:09.003557 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 13 15:54:09.003557 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 13 15:54:09.003557 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application Sep 13 15:54:09.003557 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 283/561] 10.222.0.50 () {56 vars in 1102 bytes} [Tue Sep 13 15:54:08 2022] GET /identity/v3/roles/service => generated 85 bytes in 187 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:54:09.003557 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20144948-b591-4047-94b8-ba8228d40ecc None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:09.026389 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16f3871d-3cb4-45d9-bbf2-3d74171437d4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:09.027068 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16f3871d-3cb4-45d9-bbf2-3d74171437d4 None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:09.027351 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16f3871d-3cb4-45d9-bbf2-3d74171437d4 None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:09.027627 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16f3871d-3cb4-45d9-bbf2-3d74171437d4 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:09.038596 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16f3871d-3cb4-45d9-bbf2-3d74171437d4 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:09.042752 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16f3871d-3cb4-45d9-bbf2-3d74171437d4 None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:09.059964 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application [None req-16f3871d-3cb4-45d9-bbf2-3d74171437d4 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:54:09.059964 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:54:09.059964 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:54:09.059964 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:54:09.059964 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:54:09.059964 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:54:09.059964 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:54:09.059964 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:54:09.059964 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:54:09.059964 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:54:09.059964 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:54:09.059964 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:54:09.059964 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 13 15:54:09.059964 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 13 15:54:09.059964 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 13 15:54:09.059964 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 13 15:54:09.060984 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:54:09.060984 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:54:09.060984 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:54:09.060984 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:54:09.060984 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:54:09.060984 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:54:09.060984 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:54:09.060984 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application with Lock( Sep 13 15:54:09.060984 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:54:09.060984 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:54:09.060984 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:54:09.060984 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:54:09.060984 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:54:09.060984 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:54:09.060984 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:54:09.060984 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:54:09.060984 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 13 15:54:09.062038 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 13 15:54:09.062038 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:54:09.062038 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Sep 13 15:54:09.065936 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 279/562] 10.222.0.50 () {56 vars in 1108 bytes} [Tue Sep 13 15:54:08 2022] GET /identity/v3/domains/Default => generated 87 bytes in 212 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:54:09.066802 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16f3871d-3cb4-45d9-bbf2-3d74171437d4 None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:09.112252 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20144948-b591-4047-94b8-ba8228d40ecc None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:09.146464 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c685a0c-0a04-4a75-bc5a-8fe62b5c6665 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:09.147118 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c685a0c-0a04-4a75-bc5a-8fe62b5c6665 None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:09.147397 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c685a0c-0a04-4a75-bc5a-8fe62b5c6665 None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:09.147679 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c685a0c-0a04-4a75-bc5a-8fe62b5c6665 None admin] PATH_INFO: `/v3/users/nova` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:09.154623 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c685a0c-0a04-4a75-bc5a-8fe62b5c6665 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:09.156117 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c685a0c-0a04-4a75-bc5a-8fe62b5c6665 None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:09.157826 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:54:09.168685 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16f3871d-3cb4-45d9-bbf2-3d74171437d4 None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:09.169272 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application [None req-1c685a0c-0a04-4a75-bc5a-8fe62b5c6665 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Sep 13 15:54:09.169272 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:54:09.169272 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:54:09.169272 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:54:09.169272 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:54:09.169272 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:54:09.169272 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:54:09.169272 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:54:09.169272 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:54:09.169272 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:54:09.169272 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:54:09.169272 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:54:09.169272 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 13 15:54:09.169272 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 13 15:54:09.169272 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 13 15:54:09.169272 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 13 15:54:09.170460 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:54:09.170460 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:54:09.170460 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 13 15:54:09.170460 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 13 15:54:09.170460 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 13 15:54:09.170460 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 13 15:54:09.170460 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:54:09.170460 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:54:09.170460 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:54:09.170460 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:54:09.170460 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:54:09.170460 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application with Lock( Sep 13 15:54:09.170460 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:54:09.170460 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:54:09.170460 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:54:09.170460 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:54:09.170460 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:54:09.170460 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:54:09.171633 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:54:09.171633 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:54:09.171633 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 13 15:54:09.171633 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 13 15:54:09.171633 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 13 15:54:09.171633 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 13 15:54:09.171633 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 13 15:54:09.171633 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 13 15:54:09.171633 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Sep 13 15:54:09.171633 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application Sep 13 15:54:09.171633 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 284/563] 10.222.0.50 () {56 vars in 1093 bytes} [Tue Sep 13 15:54:09 2022] GET /identity/v3/users/nova => generated 82 bytes in 167 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:54:09.171633 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c685a0c-0a04-4a75-bc5a-8fe62b5c6665 None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:09.180612 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:54:09.206868 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c0b0aa8-36cf-4747-95e2-4fb124c9910d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:09.207528 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c0b0aa8-36cf-4747-95e2-4fb124c9910d None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:09.207812 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c0b0aa8-36cf-4747-95e2-4fb124c9910d None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:09.208087 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c0b0aa8-36cf-4747-95e2-4fb124c9910d None admin] PATH_INFO: `/v3/roles/service` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:09.219017 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c0b0aa8-36cf-4747-95e2-4fb124c9910d None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 13 15:54:09.224257 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application [None req-7c0b0aa8-36cf-4747-95e2-4fb124c9910d None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 13 15:54:09.224257 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:54:09.224257 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:54:09.224257 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:54:09.224257 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:54:09.224257 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:54:09.224257 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:54:09.224257 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:54:09.224257 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:54:09.224257 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:54:09.224257 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:54:09.224257 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:54:09.224257 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 13 15:54:09.224257 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 13 15:54:09.224257 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 13 15:54:09.224257 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 13 15:54:09.225193 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 13 15:54:09.225193 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 13 15:54:09.225193 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 13 15:54:09.225193 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 13 15:54:09.225193 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:54:09.225193 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:54:09.225193 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:54:09.225193 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:54:09.225193 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:54:09.225193 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:54:09.225193 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:54:09.225193 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application with Lock( Sep 13 15:54:09.225193 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:54:09.225193 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:54:09.225193 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:54:09.225193 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:54:09.226035 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:54:09.226035 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:54:09.226035 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:54:09.226035 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:54:09.226035 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 13 15:54:09.226035 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 13 15:54:09.226035 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 13 15:54:09.226035 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 13 15:54:09.226035 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 13 15:54:09.226035 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 13 15:54:09.226035 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 13 15:54:09.226035 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Sep 13 15:54:09.226035 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 280/564] 10.222.0.50 () {56 vars in 1102 bytes} [Tue Sep 13 15:54:09 2022] GET /identity/v3/roles/service => generated 85 bytes in 159 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:54:09.226779 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c0b0aa8-36cf-4747-95e2-4fb124c9910d None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:09.276303 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c685a0c-0a04-4a75-bc5a-8fe62b5c6665 None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:09.310279 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70839c66-7944-4342-98e9-b600a4382078 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:09.310878 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70839c66-7944-4342-98e9-b600a4382078 None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:09.311182 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70839c66-7944-4342-98e9-b600a4382078 None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:09.311451 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70839c66-7944-4342-98e9-b600a4382078 None admin] PATH_INFO: `/v3/roles` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:09.312102 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70839c66-7944-4342-98e9-b600a4382078 None admin] RBAC: Adding query filter params (name=service) {{(pid=81437) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:54:09.312407 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70839c66-7944-4342-98e9-b600a4382078 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:09.313816 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70839c66-7944-4342-98e9-b600a4382078 None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:09.321465 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 285/565] 10.222.0.50 () {56 vars in 1103 bytes} [Tue Sep 13 15:54:09 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:54:09.322979 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ef1c76c-83d9-4ccb-bc29-dfd862efd31c None None] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:09.323260 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ef1c76c-83d9-4ccb-bc29-dfd862efd31c None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:09.323531 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ef1c76c-83d9-4ccb-bc29-dfd862efd31c None None] PATH_INFO: `/` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:09.324716 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 286/566] 10.222.0.50 () {54 vars in 927 bytes} [Tue Sep 13 15:54:09 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:54:09.325408 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ef1c76c-83d9-4ccb-bc29-dfd862efd31c None None] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:09.329340 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c0b0aa8-36cf-4747-95e2-4fb124c9910d None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:09.364518 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e19a1b7-f1a0-4a5f-9705-0c37b515be62 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:09.365120 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e19a1b7-f1a0-4a5f-9705-0c37b515be62 None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:09.365399 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e19a1b7-f1a0-4a5f-9705-0c37b515be62 None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:09.365670 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e19a1b7-f1a0-4a5f-9705-0c37b515be62 None admin] PATH_INFO: `/v3/domains` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:09.366639 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e19a1b7-f1a0-4a5f-9705-0c37b515be62 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81438) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:54:09.366639 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e19a1b7-f1a0-4a5f-9705-0c37b515be62 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:09.367981 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e19a1b7-f1a0-4a5f-9705-0c37b515be62 None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:09.379720 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 281/567] 10.222.0.50 () {56 vars in 1109 bytes} [Tue Sep 13 15:54:09 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:54:09.380504 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e19a1b7-f1a0-4a5f-9705-0c37b515be62 None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:09.385190 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:54:09.408429 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:54:09.428672 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ef1c76c-83d9-4ccb-bc29-dfd862efd31c None None] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:09.461570 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01749a37-e2ce-4098-89fc-ecaaa86d10cf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:09.462162 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01749a37-e2ce-4098-89fc-ecaaa86d10cf None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:09.462439 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01749a37-e2ce-4098-89fc-ecaaa86d10cf None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:09.463439 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01749a37-e2ce-4098-89fc-ecaaa86d10cf None admin] PATH_INFO: `/v3/users` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:09.464390 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01749a37-e2ce-4098-89fc-ecaaa86d10cf None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81437) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:54:09.464694 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01749a37-e2ce-4098-89fc-ecaaa86d10cf None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:09.466110 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01749a37-e2ce-4098-89fc-ecaaa86d10cf None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:09.480582 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e19a1b7-f1a0-4a5f-9705-0c37b515be62 None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:09.483561 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 287/568] 10.222.0.50 () {56 vars in 1133 bytes} [Tue Sep 13 15:54:09 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:54:09.484360 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01749a37-e2ce-4098-89fc-ecaaa86d10cf None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:09.514255 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53fe9af3-1105-4cb7-be50-a5b03c25d9c5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:09.514864 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53fe9af3-1105-4cb7-be50-a5b03c25d9c5 None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:09.515175 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53fe9af3-1105-4cb7-be50-a5b03c25d9c5 None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:09.515448 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53fe9af3-1105-4cb7-be50-a5b03c25d9c5 None admin] PATH_INFO: `/v3/roles` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:09.516104 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53fe9af3-1105-4cb7-be50-a5b03c25d9c5 None admin] RBAC: Adding query filter params (name=service) {{(pid=81438) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:54:09.516408 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53fe9af3-1105-4cb7-be50-a5b03c25d9c5 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:09.517760 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53fe9af3-1105-4cb7-be50-a5b03c25d9c5 None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:09.526017 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 282/569] 10.222.0.50 () {56 vars in 1103 bytes} [Tue Sep 13 15:54:09 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:54:09.527279 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbe49b67-7624-4e04-afc5-1703458d46f5 None None] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:09.527570 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbe49b67-7624-4e04-afc5-1703458d46f5 None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:09.527852 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbe49b67-7624-4e04-afc5-1703458d46f5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:09.545545 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.auth.core [None req-bbe49b67-7624-4e04-afc5-1703458d46f5 None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81438) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:54:09.546738 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-bbe49b67-7624-4e04-afc5-1703458d46f5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:54:09.584078 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01749a37-e2ce-4098-89fc-ecaaa86d10cf None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:09.601249 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:54:09.626926 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 283/570] 10.222.0.50 () {58 vars in 1024 bytes} [Tue Sep 13 15:54:09 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 13 15:54:09.626926 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbe49b67-7624-4e04-afc5-1703458d46f5 None None] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:09.632672 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5203483e-1c48-4e27-a97c-15c242e243e8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:09.633300 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5203483e-1c48-4e27-a97c-15c242e243e8 None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:09.633580 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5203483e-1c48-4e27-a97c-15c242e243e8 None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:09.633851 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5203483e-1c48-4e27-a97c-15c242e243e8 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:09.635781 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:54:09.651079 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5203483e-1c48-4e27-a97c-15c242e243e8 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:09.652513 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5203483e-1c48-4e27-a97c-15c242e243e8 None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:09.667695 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application [None req-5203483e-1c48-4e27-a97c-15c242e243e8 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:54:09.667695 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:54:09.667695 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:54:09.667695 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:54:09.667695 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:54:09.667695 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:54:09.667695 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:54:09.667695 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:54:09.667695 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:54:09.667695 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:54:09.667695 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:54:09.667695 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:54:09.667695 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 13 15:54:09.667695 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 13 15:54:09.667695 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 13 15:54:09.667695 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 13 15:54:09.668649 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:54:09.668649 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:54:09.668649 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:54:09.668649 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:54:09.668649 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:54:09.668649 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:54:09.668649 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:54:09.668649 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application with Lock( Sep 13 15:54:09.668649 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:54:09.668649 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:54:09.668649 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:54:09.668649 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:54:09.668649 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:54:09.668649 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:54:09.668649 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:54:09.668649 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:54:09.668649 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 13 15:54:09.669566 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 13 15:54:09.669566 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:54:09.669566 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application Sep 13 15:54:09.673477 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 288/571] 10.222.0.50 () {56 vars in 1108 bytes} [Tue Sep 13 15:54:09 2022] GET /identity/v3/domains/Default => generated 87 bytes in 189 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:54:09.674313 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5203483e-1c48-4e27-a97c-15c242e243e8 None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:09.721445 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbe49b67-7624-4e04-afc5-1703458d46f5 None None] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:09.760739 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ebc9e9c-c370-443d-b102-207cdba697a9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:09.761508 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ebc9e9c-c370-443d-b102-207cdba697a9 None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:09.761888 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ebc9e9c-c370-443d-b102-207cdba697a9 None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:09.762260 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ebc9e9c-c370-443d-b102-207cdba697a9 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:09.769628 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ebc9e9c-c370-443d-b102-207cdba697a9 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:09.771252 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ebc9e9c-c370-443d-b102-207cdba697a9 None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:09.784354 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application [None req-1ebc9e9c-c370-443d-b102-207cdba697a9 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 13 15:54:09.784354 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:54:09.784354 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:54:09.784354 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:54:09.784354 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:54:09.784354 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:54:09.784354 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:54:09.784354 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:54:09.784354 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:54:09.784354 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:54:09.784354 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:54:09.784354 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:54:09.784354 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 13 15:54:09.784354 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 13 15:54:09.784354 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 13 15:54:09.785483 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 13 15:54:09.785483 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:54:09.785483 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:54:09.785483 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:54:09.785483 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:54:09.785483 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:54:09.785483 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:54:09.785483 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:54:09.785483 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application with Lock( Sep 13 15:54:09.785483 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:54:09.785483 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:54:09.785483 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:54:09.785483 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:54:09.785483 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:54:09.785483 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:54:09.785483 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:54:09.785483 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:54:09.786592 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 13 15:54:09.786592 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 13 15:54:09.786592 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 13 15:54:09.786592 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 13 15:54:09.786592 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 13 15:54:09.786592 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 13 15:54:09.786592 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 13 15:54:09.786592 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Sep 13 15:54:09.787162 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5203483e-1c48-4e27-a97c-15c242e243e8 None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:09.788396 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 284/572] 10.222.0.50 () {56 vars in 1111 bytes} [Tue Sep 13 15:54:09 2022] GET /identity/v3/projects/service => generated 88 bytes in 175 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:54:09.789301 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ebc9e9c-c370-443d-b102-207cdba697a9 None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:09.820909 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-212f2465-272a-4eee-9679-32f9d7216c11 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:09.821542 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-212f2465-272a-4eee-9679-32f9d7216c11 None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:09.821822 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-212f2465-272a-4eee-9679-32f9d7216c11 None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:09.822200 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-212f2465-272a-4eee-9679-32f9d7216c11 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:09.832262 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-212f2465-272a-4eee-9679-32f9d7216c11 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:09.834043 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-212f2465-272a-4eee-9679-32f9d7216c11 None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:09.849598 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application [None req-212f2465-272a-4eee-9679-32f9d7216c11 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:54:09.849598 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:54:09.849598 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:54:09.849598 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:54:09.849598 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:54:09.849598 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:54:09.849598 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:54:09.849598 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:54:09.849598 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:54:09.849598 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:54:09.849598 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:54:09.849598 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:54:09.849598 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 13 15:54:09.849598 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 13 15:54:09.849598 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 13 15:54:09.849598 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 13 15:54:09.850612 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:54:09.850612 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:54:09.850612 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:54:09.850612 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:54:09.850612 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:54:09.850612 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:54:09.850612 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:54:09.850612 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application with Lock( Sep 13 15:54:09.850612 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:54:09.850612 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:54:09.850612 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:54:09.850612 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:54:09.850612 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:54:09.850612 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:54:09.850612 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:54:09.850612 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:54:09.850612 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 13 15:54:09.851544 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 13 15:54:09.851544 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:54:09.851544 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application Sep 13 15:54:09.851544 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 289/573] 10.222.0.50 () {56 vars in 1108 bytes} [Tue Sep 13 15:54:09 2022] GET /identity/v3/domains/Default => generated 87 bytes in 177 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:54:09.851544 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-212f2465-272a-4eee-9679-32f9d7216c11 None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:09.879361 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:54:09.885329 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:54:09.905598 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ebc9e9c-c370-443d-b102-207cdba697a9 None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:09.940103 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b20d94ce-3a1d-4e8e-9e84-ff4a3cc95edd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:09.940731 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b20d94ce-3a1d-4e8e-9e84-ff4a3cc95edd None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:09.941013 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b20d94ce-3a1d-4e8e-9e84-ff4a3cc95edd None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:09.941289 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b20d94ce-3a1d-4e8e-9e84-ff4a3cc95edd None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:09.951599 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b20d94ce-3a1d-4e8e-9e84-ff4a3cc95edd None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:09.953035 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b20d94ce-3a1d-4e8e-9e84-ff4a3cc95edd None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:09.957091 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-212f2465-272a-4eee-9679-32f9d7216c11 None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:09.967982 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application [None req-b20d94ce-3a1d-4e8e-9e84-ff4a3cc95edd None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:54:09.967982 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:54:09.967982 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:54:09.967982 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:54:09.967982 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:54:09.967982 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:54:09.967982 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:54:09.967982 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:54:09.967982 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:54:09.967982 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:54:09.967982 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:54:09.967982 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:54:09.967982 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 13 15:54:09.967982 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 13 15:54:09.967982 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 13 15:54:09.967982 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 13 15:54:09.969126 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:54:09.969126 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:54:09.969126 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:54:09.969126 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:54:09.969126 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:54:09.969126 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:54:09.969126 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:54:09.969126 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application with Lock( Sep 13 15:54:09.969126 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:54:09.969126 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:54:09.969126 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:54:09.969126 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:54:09.969126 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:54:09.969126 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:54:09.969126 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:54:09.969126 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:54:09.969126 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 13 15:54:09.970030 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 13 15:54:09.970030 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:54:09.970030 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Sep 13 15:54:09.970030 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 285/574] 10.222.0.50 () {56 vars in 1108 bytes} [Tue Sep 13 15:54:09 2022] GET /identity/v3/domains/Default => generated 87 bytes in 180 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:54:09.970257 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b20d94ce-3a1d-4e8e-9e84-ff4a3cc95edd None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:09.992343 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f3cfca2-5488-4d94-b8c3-ea3d1cbe87b9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:09.992878 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f3cfca2-5488-4d94-b8c3-ea3d1cbe87b9 None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:09.993153 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f3cfca2-5488-4d94-b8c3-ea3d1cbe87b9 None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:09.993423 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f3cfca2-5488-4d94-b8c3-ea3d1cbe87b9 None admin] PATH_INFO: `/v3/domains` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:09.994198 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f3cfca2-5488-4d94-b8c3-ea3d1cbe87b9 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81437) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:54:09.994499 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f3cfca2-5488-4d94-b8c3-ea3d1cbe87b9 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:09.995932 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f3cfca2-5488-4d94-b8c3-ea3d1cbe87b9 None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:10.007571 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 290/575] 10.222.0.50 () {56 vars in 1109 bytes} [Tue Sep 13 15:54:09 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:54:10.011762 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f3cfca2-5488-4d94-b8c3-ea3d1cbe87b9 None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:10.073573 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b20d94ce-3a1d-4e8e-9e84-ff4a3cc95edd None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:10.093875 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:54:10.102858 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:54:10.108788 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4028e136-4f87-4dc6-9327-e83b1ddef86c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:10.109396 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4028e136-4f87-4dc6-9327-e83b1ddef86c None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:10.109680 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4028e136-4f87-4dc6-9327-e83b1ddef86c None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:10.109964 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4028e136-4f87-4dc6-9327-e83b1ddef86c None admin] PATH_INFO: `/v3/projects` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:10.110655 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4028e136-4f87-4dc6-9327-e83b1ddef86c None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81438) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:54:10.110992 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4028e136-4f87-4dc6-9327-e83b1ddef86c None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:10.112419 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4028e136-4f87-4dc6-9327-e83b1ddef86c None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:10.112740 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f3cfca2-5488-4d94-b8c3-ea3d1cbe87b9 None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:10.124522 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 286/576] 10.222.0.50 () {56 vars in 1148 bytes} [Tue Sep 13 15:54:09 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:54:10.125347 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4028e136-4f87-4dc6-9327-e83b1ddef86c None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:10.153641 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddbd46c7-3315-4f4b-bb01-7374312311ba None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:10.154244 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddbd46c7-3315-4f4b-bb01-7374312311ba None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:10.154523 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddbd46c7-3315-4f4b-bb01-7374312311ba None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:10.154795 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddbd46c7-3315-4f4b-bb01-7374312311ba None admin] PATH_INFO: `/v3/domains` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:10.155463 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ddbd46c7-3315-4f4b-bb01-7374312311ba None admin] RBAC: Adding query filter params (name=Default) {{(pid=81437) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:54:10.155767 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ddbd46c7-3315-4f4b-bb01-7374312311ba None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:10.157133 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ddbd46c7-3315-4f4b-bb01-7374312311ba None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:10.169042 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 291/577] 10.222.0.50 () {56 vars in 1109 bytes} [Tue Sep 13 15:54:10 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:54:10.169663 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddbd46c7-3315-4f4b-bb01-7374312311ba None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:10.225309 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4028e136-4f87-4dc6-9327-e83b1ddef86c None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:10.260698 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfcd7769-de3a-4431-8244-fcec395801d2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:10.261263 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfcd7769-de3a-4431-8244-fcec395801d2 None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:10.261545 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfcd7769-de3a-4431-8244-fcec395801d2 None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:10.261820 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfcd7769-de3a-4431-8244-fcec395801d2 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:10.266753 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddbd46c7-3315-4f4b-bb01-7374312311ba None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:10.272050 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dfcd7769-de3a-4431-8244-fcec395801d2 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 13 15:54:10.277367 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application [None req-dfcd7769-de3a-4431-8244-fcec395801d2 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 13 15:54:10.277367 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:54:10.277367 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:54:10.277367 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:54:10.277367 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:54:10.277367 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:54:10.277367 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:54:10.277367 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:54:10.277367 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:54:10.277367 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:54:10.277367 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:54:10.277367 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:54:10.277367 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 13 15:54:10.277367 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 13 15:54:10.277367 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 13 15:54:10.277367 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 13 15:54:10.278322 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 13 15:54:10.278322 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 13 15:54:10.278322 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 13 15:54:10.278322 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 13 15:54:10.278322 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:54:10.278322 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:54:10.278322 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:54:10.278322 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:54:10.278322 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:54:10.278322 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:54:10.278322 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:54:10.278322 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application with Lock( Sep 13 15:54:10.278322 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:54:10.278322 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:54:10.278322 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:54:10.278322 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:54:10.279225 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:54:10.279225 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:54:10.279225 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:54:10.279225 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:54:10.279225 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 13 15:54:10.279225 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 13 15:54:10.279225 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 13 15:54:10.279225 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 13 15:54:10.279225 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 13 15:54:10.279225 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 13 15:54:10.279225 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 13 15:54:10.279225 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Sep 13 15:54:10.279225 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 287/578] 10.222.0.50 () {56 vars in 1102 bytes} [Tue Sep 13 15:54:10 2022] GET /identity/v3/roles/service => generated 85 bytes in 153 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:54:10.279225 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfcd7769-de3a-4431-8244-fcec395801d2 None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:10.303964 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-976afbf2-f2d4-4096-a95c-d0de364a12c2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:10.304559 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-976afbf2-f2d4-4096-a95c-d0de364a12c2 None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:10.304848 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-976afbf2-f2d4-4096-a95c-d0de364a12c2 None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:10.305124 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-976afbf2-f2d4-4096-a95c-d0de364a12c2 None admin] PATH_INFO: `/v3/domains` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:10.305755 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-976afbf2-f2d4-4096-a95c-d0de364a12c2 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81437) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:54:10.306088 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-976afbf2-f2d4-4096-a95c-d0de364a12c2 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:10.307564 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-976afbf2-f2d4-4096-a95c-d0de364a12c2 None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:10.319227 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 292/579] 10.222.0.50 () {56 vars in 1109 bytes} [Tue Sep 13 15:54:10 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:54:10.320010 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-976afbf2-f2d4-4096-a95c-d0de364a12c2 None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:10.323793 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:54:10.335627 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:54:10.381583 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfcd7769-de3a-4431-8244-fcec395801d2 None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:10.416982 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bae6756e-aaf8-4025-b05c-326f491b506b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:10.417647 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bae6756e-aaf8-4025-b05c-326f491b506b None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:10.417932 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bae6756e-aaf8-4025-b05c-326f491b506b None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:10.418203 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bae6756e-aaf8-4025-b05c-326f491b506b None admin] PATH_INFO: `/v3/users/cinder` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:10.421303 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-976afbf2-f2d4-4096-a95c-d0de364a12c2 None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:10.425088 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bae6756e-aaf8-4025-b05c-326f491b506b None admin] RBAC: Authorizing `identity:get_user(user_id=cinder)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:10.426542 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bae6756e-aaf8-4025-b05c-326f491b506b None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:10.438102 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application [None req-bae6756e-aaf8-4025-b05c-326f491b506b None admin] Could not find user: cinder.: keystone.exception.UserNotFound: Could not find user: cinder. Sep 13 15:54:10.438102 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:54:10.438102 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:54:10.438102 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:54:10.438102 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:54:10.438102 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:54:10.438102 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:54:10.438102 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:54:10.438102 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:54:10.438102 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:54:10.438102 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:54:10.438102 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:54:10.438102 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 13 15:54:10.438102 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 13 15:54:10.438102 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 13 15:54:10.438102 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 13 15:54:10.439249 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:54:10.439249 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:54:10.439249 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 13 15:54:10.439249 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 13 15:54:10.439249 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 13 15:54:10.439249 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 13 15:54:10.439249 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:54:10.439249 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:54:10.439249 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:54:10.439249 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:54:10.439249 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:54:10.439249 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application with Lock( Sep 13 15:54:10.439249 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:54:10.439249 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:54:10.439249 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:54:10.439249 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:54:10.439249 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:54:10.439249 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:54:10.440200 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:54:10.440200 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:54:10.440200 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 13 15:54:10.440200 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 13 15:54:10.440200 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 13 15:54:10.440200 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 13 15:54:10.440200 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 13 15:54:10.440200 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 13 15:54:10.440200 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: cinder. Sep 13 15:54:10.440200 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Sep 13 15:54:10.440200 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 288/580] 10.222.0.50 () {56 vars in 1099 bytes} [Tue Sep 13 15:54:10 2022] GET /identity/v3/users/cinder => generated 84 bytes in 160 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:54:10.440780 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bae6756e-aaf8-4025-b05c-326f491b506b None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:10.458876 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1872848-1e35-43fa-aaa7-0d7b53ad67ee None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:10.459501 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1872848-1e35-43fa-aaa7-0d7b53ad67ee None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:10.459772 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1872848-1e35-43fa-aaa7-0d7b53ad67ee None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:10.460038 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1872848-1e35-43fa-aaa7-0d7b53ad67ee None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:10.460749 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1872848-1e35-43fa-aaa7-0d7b53ad67ee None admin] RBAC: Adding query filter params (role.id=da847124d9374ac88b27968451c486ab, scope.project.id=5811e3b83a1e462e9320cad79a3ad791, user.id=b00ab0d6321d483188bc69deaddcdbbc) {{(pid=81437) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:54:10.461054 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1872848-1e35-43fa-aaa7-0d7b53ad67ee None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:10.462499 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1872848-1e35-43fa-aaa7-0d7b53ad67ee None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:10.468633 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 293/581] 10.222.0.50 () {56 vars in 1374 bytes} [Tue Sep 13 15:54:10 2022] GET /identity/v3/role_assignments?user.id=b00ab0d6321d483188bc69deaddcdbbc&scope.project.id=5811e3b83a1e462e9320cad79a3ad791&role.id=da847124d9374ac88b27968451c486ab => generated 259 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:54:10.469391 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1872848-1e35-43fa-aaa7-0d7b53ad67ee None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:10.548102 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bae6756e-aaf8-4025-b05c-326f491b506b None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:10.563697 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:54:10.570781 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:54:10.588794 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff40fb39-af70-40d7-981c-1cc09fba10f4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:10.589457 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff40fb39-af70-40d7-981c-1cc09fba10f4 None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:10.590061 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff40fb39-af70-40d7-981c-1cc09fba10f4 None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:10.590341 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff40fb39-af70-40d7-981c-1cc09fba10f4 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:10.598814 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff40fb39-af70-40d7-981c-1cc09fba10f4 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:10.600831 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff40fb39-af70-40d7-981c-1cc09fba10f4 None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:10.613717 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1872848-1e35-43fa-aaa7-0d7b53ad67ee None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:10.614255 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application [None req-ff40fb39-af70-40d7-981c-1cc09fba10f4 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 13 15:54:10.614255 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:54:10.614255 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:54:10.614255 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:54:10.614255 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:54:10.614255 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:54:10.614255 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:54:10.614255 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:54:10.614255 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:54:10.614255 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:54:10.614255 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:54:10.614255 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:54:10.614255 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 13 15:54:10.614255 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 13 15:54:10.614255 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 13 15:54:10.615154 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 13 15:54:10.615154 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:54:10.615154 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:54:10.615154 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:54:10.615154 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:54:10.615154 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:54:10.615154 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:54:10.615154 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:54:10.615154 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application with Lock( Sep 13 15:54:10.615154 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:54:10.615154 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:54:10.615154 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:54:10.615154 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:54:10.615154 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:54:10.615154 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:54:10.615154 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:54:10.615154 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:54:10.616075 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 13 15:54:10.616075 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 13 15:54:10.616075 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 13 15:54:10.616075 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 13 15:54:10.616075 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 13 15:54:10.616075 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 13 15:54:10.616075 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 13 15:54:10.616075 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Sep 13 15:54:10.616075 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 289/582] 10.222.0.50 () {56 vars in 1111 bytes} [Tue Sep 13 15:54:10 2022] GET /identity/v3/projects/service => generated 88 bytes in 175 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:54:10.616575 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff40fb39-af70-40d7-981c-1cc09fba10f4 None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:10.658956 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c62753df-c7b4-4818-8298-b4ef5b32e4a1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:10.660161 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c62753df-c7b4-4818-8298-b4ef5b32e4a1 None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:10.660439 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c62753df-c7b4-4818-8298-b4ef5b32e4a1 None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:10.660711 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c62753df-c7b4-4818-8298-b4ef5b32e4a1 None admin] PATH_INFO: `/v3/roles` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:10.661366 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c62753df-c7b4-4818-8298-b4ef5b32e4a1 None admin] RBAC: Adding query filter params (name=service) {{(pid=81437) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:54:10.661672 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c62753df-c7b4-4818-8298-b4ef5b32e4a1 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:10.663577 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c62753df-c7b4-4818-8298-b4ef5b32e4a1 None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:10.676102 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 294/583] 10.222.0.50 () {56 vars in 1103 bytes} [Tue Sep 13 15:54:10 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 203 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:54:10.676102 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c62753df-c7b4-4818-8298-b4ef5b32e4a1 None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:10.742071 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff40fb39-af70-40d7-981c-1cc09fba10f4 None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:10.777061 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c62753df-c7b4-4818-8298-b4ef5b32e4a1 None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:10.783342 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b778d4c-70b2-4d22-a665-cb4a6449f132 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:10.783972 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b778d4c-70b2-4d22-a665-cb4a6449f132 None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:10.784248 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b778d4c-70b2-4d22-a665-cb4a6449f132 None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:10.784520 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b778d4c-70b2-4d22-a665-cb4a6449f132 None admin] PATH_INFO: `/v3/users/glance` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:10.794431 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b778d4c-70b2-4d22-a665-cb4a6449f132 None admin] RBAC: Authorizing `identity:get_user(user_id=glance)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:10.795905 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b778d4c-70b2-4d22-a665-cb4a6449f132 None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:10.805518 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:54:10.808675 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application [None req-7b778d4c-70b2-4d22-a665-cb4a6449f132 None admin] Could not find user: glance.: keystone.exception.UserNotFound: Could not find user: glance. Sep 13 15:54:10.808675 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:54:10.808675 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:54:10.808675 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:54:10.808675 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:54:10.808675 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:54:10.808675 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:54:10.808675 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:54:10.808675 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:54:10.808675 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:54:10.808675 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:54:10.808675 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:54:10.808675 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 13 15:54:10.808675 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 13 15:54:10.808675 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 13 15:54:10.808675 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 13 15:54:10.809619 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:54:10.809619 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:54:10.809619 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 13 15:54:10.809619 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 13 15:54:10.809619 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 13 15:54:10.809619 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 13 15:54:10.809619 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:54:10.809619 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:54:10.809619 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:54:10.809619 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:54:10.809619 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:54:10.809619 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application with Lock( Sep 13 15:54:10.809619 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:54:10.809619 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:54:10.809619 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:54:10.809619 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:54:10.809619 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:54:10.809619 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:54:10.811410 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:54:10.811410 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:54:10.811410 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 13 15:54:10.811410 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 13 15:54:10.811410 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 13 15:54:10.811410 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 13 15:54:10.811410 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 13 15:54:10.811410 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 13 15:54:10.811410 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance. Sep 13 15:54:10.811410 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Sep 13 15:54:10.811410 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 290/584] 10.222.0.50 () {56 vars in 1099 bytes} [Tue Sep 13 15:54:10 2022] GET /identity/v3/users/glance => generated 84 bytes in 194 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:54:10.811410 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b778d4c-70b2-4d22-a665-cb4a6449f132 None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:10.819967 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65234425-8748-4815-9a30-1183f4636a0d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:10.820696 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65234425-8748-4815-9a30-1183f4636a0d None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:10.820978 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65234425-8748-4815-9a30-1183f4636a0d None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:10.821251 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65234425-8748-4815-9a30-1183f4636a0d None admin] PATH_INFO: `/v3/users` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:10.822005 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65234425-8748-4815-9a30-1183f4636a0d None admin] RBAC: Adding query filter params (domain_id=default, name=cinder) {{(pid=81437) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:54:10.822329 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65234425-8748-4815-9a30-1183f4636a0d None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:10.823933 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65234425-8748-4815-9a30-1183f4636a0d None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:10.843940 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 295/585] 10.222.0.50 () {56 vars in 1137 bytes} [Tue Sep 13 15:54:10 2022] GET /identity/v3/users?domain_id=default&name=cinder => generated 372 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:54:10.844784 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65234425-8748-4815-9a30-1183f4636a0d None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:10.849239 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:54:10.924247 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b778d4c-70b2-4d22-a665-cb4a6449f132 None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:10.952994 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65234425-8748-4815-9a30-1183f4636a0d None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:10.962359 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eaf47249-5f1a-4183-ba25-b3dfc63d58ce None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:10.963776 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaf47249-5f1a-4183-ba25-b3dfc63d58ce None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:10.964056 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaf47249-5f1a-4183-ba25-b3dfc63d58ce None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:10.964330 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaf47249-5f1a-4183-ba25-b3dfc63d58ce None admin] PATH_INFO: `/v3/projects` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:10.965019 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eaf47249-5f1a-4183-ba25-b3dfc63d58ce None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81438) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:54:10.965324 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eaf47249-5f1a-4183-ba25-b3dfc63d58ce None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:10.966759 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eaf47249-5f1a-4183-ba25-b3dfc63d58ce None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:10.979829 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 291/586] 10.222.0.50 () {56 vars in 1148 bytes} [Tue Sep 13 15:54:10 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:54:10.980651 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eaf47249-5f1a-4183-ba25-b3dfc63d58ce None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:10.998556 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e4f52f8-c26d-4a8f-870b-76dc0eea0c09 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:10.999464 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e4f52f8-c26d-4a8f-870b-76dc0eea0c09 None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:10.999885 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e4f52f8-c26d-4a8f-870b-76dc0eea0c09 None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:11.000288 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e4f52f8-c26d-4a8f-870b-76dc0eea0c09 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:11.011373 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e4f52f8-c26d-4a8f-870b-76dc0eea0c09 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:11.013138 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e4f52f8-c26d-4a8f-870b-76dc0eea0c09 None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:11.028650 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application [None req-9e4f52f8-c26d-4a8f-870b-76dc0eea0c09 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:54:11.028650 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:54:11.028650 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:54:11.028650 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:54:11.028650 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:54:11.028650 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:54:11.028650 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:54:11.028650 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:54:11.028650 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:54:11.028650 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:54:11.028650 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:54:11.028650 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:54:11.028650 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 13 15:54:11.028650 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 13 15:54:11.028650 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 13 15:54:11.028650 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 13 15:54:11.029793 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:54:11.029793 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:54:11.029793 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:54:11.029793 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:54:11.029793 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:54:11.029793 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:54:11.029793 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:54:11.029793 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application with Lock( Sep 13 15:54:11.029793 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:54:11.029793 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:54:11.029793 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:54:11.029793 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:54:11.029793 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:54:11.029793 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:54:11.029793 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:54:11.029793 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:54:11.029793 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 13 15:54:11.030705 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 13 15:54:11.030705 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:54:11.030705 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application Sep 13 15:54:11.030705 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 296/587] 10.222.0.50 () {56 vars in 1108 bytes} [Tue Sep 13 15:54:10 2022] GET /identity/v3/domains/Default => generated 87 bytes in 185 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:54:11.031031 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e4f52f8-c26d-4a8f-870b-76dc0eea0c09 None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:11.074462 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:54:11.096280 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:54:11.108684 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eaf47249-5f1a-4183-ba25-b3dfc63d58ce None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:11.139448 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e4f52f8-c26d-4a8f-870b-76dc0eea0c09 None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:11.144340 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c816764-fe4e-4e31-9bca-a978c0cc6ce5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:11.145030 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c816764-fe4e-4e31-9bca-a978c0cc6ce5 None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:11.145381 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c816764-fe4e-4e31-9bca-a978c0cc6ce5 None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:11.145727 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c816764-fe4e-4e31-9bca-a978c0cc6ce5 None admin] PATH_INFO: `/v3/users` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:11.146519 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c816764-fe4e-4e31-9bca-a978c0cc6ce5 None admin] RBAC: Adding query filter params (domain_id=default, name=glance) {{(pid=81438) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:54:11.146915 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c816764-fe4e-4e31-9bca-a978c0cc6ce5 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:11.148451 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c816764-fe4e-4e31-9bca-a978c0cc6ce5 None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:11.167444 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 292/588] 10.222.0.50 () {56 vars in 1137 bytes} [Tue Sep 13 15:54:10 2022] GET /identity/v3/users?domain_id=default&name=glance => generated 372 bytes in 187 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:54:11.168251 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c816764-fe4e-4e31-9bca-a978c0cc6ce5 None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:11.180016 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ba6cda9-4cd2-4c9a-a230-5e49c380a04d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:11.180643 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ba6cda9-4cd2-4c9a-a230-5e49c380a04d None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:11.180926 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ba6cda9-4cd2-4c9a-a230-5e49c380a04d None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:11.181208 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ba6cda9-4cd2-4c9a-a230-5e49c380a04d None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:11.192042 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ba6cda9-4cd2-4c9a-a230-5e49c380a04d None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:11.193522 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ba6cda9-4cd2-4c9a-a230-5e49c380a04d None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:11.209413 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application [None req-6ba6cda9-4cd2-4c9a-a230-5e49c380a04d None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:54:11.209413 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:54:11.209413 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:54:11.209413 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:54:11.209413 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:54:11.209413 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:54:11.209413 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:54:11.209413 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:54:11.209413 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:54:11.209413 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:54:11.209413 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:54:11.209413 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:54:11.209413 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 13 15:54:11.209413 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 13 15:54:11.209413 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 13 15:54:11.209413 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 13 15:54:11.210583 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:54:11.210583 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:54:11.210583 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:54:11.210583 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:54:11.210583 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:54:11.210583 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:54:11.210583 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:54:11.210583 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application with Lock( Sep 13 15:54:11.210583 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:54:11.210583 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:54:11.210583 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:54:11.210583 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:54:11.210583 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:54:11.210583 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:54:11.210583 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:54:11.210583 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:54:11.210583 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 13 15:54:11.211734 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 13 15:54:11.211734 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:54:11.211734 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application Sep 13 15:54:11.211734 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 297/589] 10.222.0.50 () {56 vars in 1108 bytes} [Tue Sep 13 15:54:11 2022] GET /identity/v3/domains/Default => generated 87 bytes in 180 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:54:11.211734 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ba6cda9-4cd2-4c9a-a230-5e49c380a04d None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:11.283068 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c816764-fe4e-4e31-9bca-a978c0cc6ce5 None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:11.304330 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:54:11.315503 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ba6cda9-4cd2-4c9a-a230-5e49c380a04d None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:11.321352 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-602e49da-8f14-4a26-8d06-937cfc7f2ff7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:11.322618 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-602e49da-8f14-4a26-8d06-937cfc7f2ff7 None admin] REQUEST_METHOD: `PUT` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:11.323648 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-602e49da-8f14-4a26-8d06-937cfc7f2ff7 None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:11.323931 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-602e49da-8f14-4a26-8d06-937cfc7f2ff7 None admin] PATH_INFO: `/v3/projects/5811e3b83a1e462e9320cad79a3ad791/users/4e039b46c5e3445bbb39904b00c5430e/roles/da847124d9374ac88b27968451c486ab` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:11.340707 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-602e49da-8f14-4a26-8d06-937cfc7f2ff7 None admin] RBAC: Authorizing `identity:create_grant(project_id=5811e3b83a1e462e9320cad79a3ad791, user_id=4e039b46c5e3445bbb39904b00c5430e, role_id=da847124d9374ac88b27968451c486ab)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:11.342217 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-602e49da-8f14-4a26-8d06-937cfc7f2ff7 None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:11.352335 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:54:11.355651 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60643e0f-47a7-41a8-a21a-a1625d8fd65c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:11.356271 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60643e0f-47a7-41a8-a21a-a1625d8fd65c None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:11.356550 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60643e0f-47a7-41a8-a21a-a1625d8fd65c None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:11.356823 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60643e0f-47a7-41a8-a21a-a1625d8fd65c None admin] PATH_INFO: `/v3/domains` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:11.357513 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60643e0f-47a7-41a8-a21a-a1625d8fd65c None admin] RBAC: Adding query filter params (name=Default) {{(pid=81437) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:54:11.357770 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60643e0f-47a7-41a8-a21a-a1625d8fd65c None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:11.359366 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60643e0f-47a7-41a8-a21a-a1625d8fd65c None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:11.363007 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 293/590] 10.222.0.50 () {58 vars in 1439 bytes} [Tue Sep 13 15:54:11 2022] PUT /identity/v3/projects/5811e3b83a1e462e9320cad79a3ad791/users/4e039b46c5e3445bbb39904b00c5430e/roles/da847124d9374ac88b27968451c486ab => generated 0 bytes in 189 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:54:11.364008 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-602e49da-8f14-4a26-8d06-937cfc7f2ff7 None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:11.387396 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 298/591] 10.222.0.50 () {56 vars in 1109 bytes} [Tue Sep 13 15:54:11 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:54:11.388837 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60643e0f-47a7-41a8-a21a-a1625d8fd65c None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:11.474773 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-602e49da-8f14-4a26-8d06-937cfc7f2ff7 None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:11.518875 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46f3779b-ac9b-4286-aec7-dd4ab0ade280 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:11.519600 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46f3779b-ac9b-4286-aec7-dd4ab0ade280 None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:11.519875 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46f3779b-ac9b-4286-aec7-dd4ab0ade280 None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:11.520132 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46f3779b-ac9b-4286-aec7-dd4ab0ade280 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:11.524673 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60643e0f-47a7-41a8-a21a-a1625d8fd65c None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:11.532067 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46f3779b-ac9b-4286-aec7-dd4ab0ade280 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:11.533557 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46f3779b-ac9b-4286-aec7-dd4ab0ade280 None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:11.549890 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application [None req-46f3779b-ac9b-4286-aec7-dd4ab0ade280 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:54:11.549890 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:54:11.549890 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:54:11.549890 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:54:11.549890 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:54:11.549890 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:54:11.549890 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:54:11.549890 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:54:11.549890 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:54:11.549890 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:54:11.549890 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:54:11.549890 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:54:11.549890 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 13 15:54:11.549890 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 13 15:54:11.549890 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 13 15:54:11.549890 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 13 15:54:11.550978 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:54:11.550978 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:54:11.550978 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:54:11.550978 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:54:11.550978 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:54:11.550978 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:54:11.550978 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:54:11.550978 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application with Lock( Sep 13 15:54:11.550978 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:54:11.550978 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:54:11.550978 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:54:11.550978 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:54:11.550978 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:54:11.550978 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:54:11.550978 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:54:11.550978 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:54:11.550978 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 13 15:54:11.551884 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 13 15:54:11.551884 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:54:11.551884 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Sep 13 15:54:11.552069 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 294/592] 10.222.0.50 () {56 vars in 1108 bytes} [Tue Sep 13 15:54:11 2022] GET /identity/v3/domains/Default => generated 87 bytes in 188 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:54:11.552916 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46f3779b-ac9b-4286-aec7-dd4ab0ade280 None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:11.569748 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6cff95d-111b-4f56-ad67-a8640a3fd40a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:11.570732 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6cff95d-111b-4f56-ad67-a8640a3fd40a None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:11.571259 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6cff95d-111b-4f56-ad67-a8640a3fd40a None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:11.571764 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6cff95d-111b-4f56-ad67-a8640a3fd40a None admin] PATH_INFO: `/v3/domains` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:11.572967 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6cff95d-111b-4f56-ad67-a8640a3fd40a None admin] RBAC: Adding query filter params (name=Default) {{(pid=81437) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:54:11.573657 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6cff95d-111b-4f56-ad67-a8640a3fd40a None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:11.575289 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6cff95d-111b-4f56-ad67-a8640a3fd40a None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:11.589374 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 299/593] 10.222.0.50 () {56 vars in 1109 bytes} [Tue Sep 13 15:54:11 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 201 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:54:11.590571 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6cff95d-111b-4f56-ad67-a8640a3fd40a None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:11.607492 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:54:11.614787 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:54:11.675583 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46f3779b-ac9b-4286-aec7-dd4ab0ade280 None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:11.714345 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6cff95d-111b-4f56-ad67-a8640a3fd40a None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:11.718698 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45fc8738-f213-454c-8706-9151aa272f61 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:11.719433 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45fc8738-f213-454c-8706-9151aa272f61 None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:11.719779 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45fc8738-f213-454c-8706-9151aa272f61 None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:11.720128 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45fc8738-f213-454c-8706-9151aa272f61 None admin] PATH_INFO: `/v3/users/neutron` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:11.730648 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45fc8738-f213-454c-8706-9151aa272f61 None admin] RBAC: Authorizing `identity:get_user(user_id=neutron)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:11.732464 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45fc8738-f213-454c-8706-9151aa272f61 None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:11.747660 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application [None req-45fc8738-f213-454c-8706-9151aa272f61 None admin] Could not find user: neutron.: keystone.exception.UserNotFound: Could not find user: neutron. Sep 13 15:54:11.747660 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:54:11.747660 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:54:11.747660 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:54:11.747660 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:54:11.747660 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:54:11.747660 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:54:11.747660 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:54:11.747660 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:54:11.747660 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:54:11.747660 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:54:11.747660 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:54:11.747660 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 13 15:54:11.747660 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 13 15:54:11.747660 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 13 15:54:11.747660 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 13 15:54:11.748656 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:54:11.748656 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:54:11.748656 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 13 15:54:11.748656 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 13 15:54:11.748656 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 13 15:54:11.748656 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 13 15:54:11.748656 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:54:11.748656 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:54:11.748656 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:54:11.748656 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:54:11.748656 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:54:11.748656 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application with Lock( Sep 13 15:54:11.748656 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:54:11.748656 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:54:11.748656 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:54:11.748656 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:54:11.748656 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:54:11.748656 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:54:11.750033 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:54:11.750033 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:54:11.750033 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 13 15:54:11.750033 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 13 15:54:11.750033 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 13 15:54:11.750033 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 13 15:54:11.750033 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 13 15:54:11.750033 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 13 15:54:11.750033 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: neutron. Sep 13 15:54:11.750033 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Sep 13 15:54:11.750970 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 295/594] 10.222.0.50 () {56 vars in 1102 bytes} [Tue Sep 13 15:54:11 2022] GET /identity/v3/users/neutron => generated 85 bytes in 196 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:54:11.752006 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45fc8738-f213-454c-8706-9151aa272f61 None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:11.768232 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db5f0ba1-a685-4457-9660-0665d32adc16 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:11.768932 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db5f0ba1-a685-4457-9660-0665d32adc16 None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:11.769278 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db5f0ba1-a685-4457-9660-0665d32adc16 None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:11.769618 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db5f0ba1-a685-4457-9660-0665d32adc16 None admin] PATH_INFO: `/v3/domains` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:11.770423 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db5f0ba1-a685-4457-9660-0665d32adc16 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81437) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:54:11.770811 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db5f0ba1-a685-4457-9660-0665d32adc16 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:11.772402 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db5f0ba1-a685-4457-9660-0665d32adc16 None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:11.795558 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 300/595] 10.222.0.50 () {56 vars in 1109 bytes} [Tue Sep 13 15:54:11 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 201 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:54:11.796880 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db5f0ba1-a685-4457-9660-0665d32adc16 None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:11.859480 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45fc8738-f213-454c-8706-9151aa272f61 None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:11.873651 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:54:11.890816 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:54:11.896261 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45d0c46d-f05a-4c13-a8cf-3383e726e387 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:11.897158 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45d0c46d-f05a-4c13-a8cf-3383e726e387 None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:11.897619 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45d0c46d-f05a-4c13-a8cf-3383e726e387 None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:11.898070 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45d0c46d-f05a-4c13-a8cf-3383e726e387 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:11.905726 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45d0c46d-f05a-4c13-a8cf-3383e726e387 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:11.907438 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45d0c46d-f05a-4c13-a8cf-3383e726e387 None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:11.919565 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application [None req-45d0c46d-f05a-4c13-a8cf-3383e726e387 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 13 15:54:11.919565 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:54:11.919565 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:54:11.919565 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:54:11.919565 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:54:11.919565 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:54:11.919565 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:54:11.919565 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:54:11.919565 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:54:11.919565 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:54:11.919565 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:54:11.919565 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:54:11.919565 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 13 15:54:11.919565 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 13 15:54:11.919565 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 13 15:54:11.920694 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 13 15:54:11.920694 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:54:11.920694 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:54:11.920694 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:54:11.920694 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:54:11.920694 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:54:11.920694 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:54:11.920694 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:54:11.920694 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application with Lock( Sep 13 15:54:11.920694 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:54:11.920694 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:54:11.920694 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:54:11.920694 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:54:11.920694 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:54:11.920694 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:54:11.920694 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:54:11.920694 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:54:11.921818 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 13 15:54:11.921818 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 13 15:54:11.921818 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 13 15:54:11.921818 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 13 15:54:11.921818 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 13 15:54:11.921818 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 13 15:54:11.921818 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 13 15:54:11.921818 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Sep 13 15:54:11.927066 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db5f0ba1-a685-4457-9660-0665d32adc16 None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:11.927294 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 296/596] 10.222.0.50 () {56 vars in 1111 bytes} [Tue Sep 13 15:54:11 2022] GET /identity/v3/projects/service => generated 88 bytes in 172 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:54:11.928208 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45d0c46d-f05a-4c13-a8cf-3383e726e387 None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:11.965632 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-716dc956-4a1d-4641-aa4f-20d29394855b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:11.966242 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-716dc956-4a1d-4641-aa4f-20d29394855b None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:11.966518 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-716dc956-4a1d-4641-aa4f-20d29394855b None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:11.966832 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-716dc956-4a1d-4641-aa4f-20d29394855b None admin] PATH_INFO: `/v3/users` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:11.967586 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-716dc956-4a1d-4641-aa4f-20d29394855b None admin] RBAC: Adding query filter params (domain_id=default, name=neutron) {{(pid=81437) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:54:11.967888 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-716dc956-4a1d-4641-aa4f-20d29394855b None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:11.969434 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-716dc956-4a1d-4641-aa4f-20d29394855b None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:11.989317 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 301/597] 10.222.0.50 () {56 vars in 1139 bytes} [Tue Sep 13 15:54:11 2022] GET /identity/v3/users?domain_id=default&name=neutron => generated 374 bytes in 193 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:54:11.995564 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-716dc956-4a1d-4641-aa4f-20d29394855b None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:12.033442 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45d0c46d-f05a-4c13-a8cf-3383e726e387 None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:12.069226 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0f9831c-4d2e-4273-8c40-012d9ed8f0ea None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:12.069990 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0f9831c-4d2e-4273-8c40-012d9ed8f0ea None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:12.070371 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0f9831c-4d2e-4273-8c40-012d9ed8f0ea None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:12.070746 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0f9831c-4d2e-4273-8c40-012d9ed8f0ea None admin] PATH_INFO: `/v3/projects/service` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:12.077881 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0f9831c-4d2e-4273-8c40-012d9ed8f0ea None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:12.079475 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0f9831c-4d2e-4273-8c40-012d9ed8f0ea None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:12.091266 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application [None req-f0f9831c-4d2e-4273-8c40-012d9ed8f0ea None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 13 15:54:12.091266 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:54:12.091266 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:54:12.091266 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:54:12.091266 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:54:12.091266 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:54:12.091266 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:54:12.091266 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:54:12.091266 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:54:12.091266 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:54:12.091266 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:54:12.091266 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:54:12.091266 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 13 15:54:12.091266 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 13 15:54:12.091266 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 13 15:54:12.092475 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 13 15:54:12.092475 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:54:12.092475 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:54:12.092475 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:54:12.092475 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:54:12.092475 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:54:12.092475 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:54:12.092475 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:54:12.092475 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application with Lock( Sep 13 15:54:12.092475 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:54:12.092475 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:54:12.092475 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:54:12.092475 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:54:12.092475 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:54:12.092475 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:54:12.092475 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:54:12.092475 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:54:12.093574 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 13 15:54:12.093574 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 13 15:54:12.093574 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 13 15:54:12.093574 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 13 15:54:12.093574 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 13 15:54:12.093574 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 13 15:54:12.093574 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 13 15:54:12.093574 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Sep 13 15:54:12.095193 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 297/598] 10.222.0.50 () {56 vars in 1111 bytes} [Tue Sep 13 15:54:11 2022] GET /identity/v3/projects/service => generated 88 bytes in 167 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:54:12.096205 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0f9831c-4d2e-4273-8c40-012d9ed8f0ea None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:12.118381 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-716dc956-4a1d-4641-aa4f-20d29394855b None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:12.129958 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:54:12.159550 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:54:12.173901 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72b6fec9-3740-4e0d-b33c-dbbb4bc10ec1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:12.175494 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72b6fec9-3740-4e0d-b33c-dbbb4bc10ec1 None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:12.179212 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72b6fec9-3740-4e0d-b33c-dbbb4bc10ec1 None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:12.179649 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72b6fec9-3740-4e0d-b33c-dbbb4bc10ec1 None admin] PATH_INFO: `/v3/projects` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:12.181126 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72b6fec9-3740-4e0d-b33c-dbbb4bc10ec1 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81437) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:54:12.182429 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72b6fec9-3740-4e0d-b33c-dbbb4bc10ec1 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:12.184931 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72b6fec9-3740-4e0d-b33c-dbbb4bc10ec1 None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:12.202929 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 302/599] 10.222.0.50 () {56 vars in 1148 bytes} [Tue Sep 13 15:54:11 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 213 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:54:12.204042 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72b6fec9-3740-4e0d-b33c-dbbb4bc10ec1 None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:12.215731 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0f9831c-4d2e-4273-8c40-012d9ed8f0ea None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:12.253168 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6979da2-e2fe-4817-9b4f-4cd05e5e93cb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:12.253917 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6979da2-e2fe-4817-9b4f-4cd05e5e93cb None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:12.254270 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6979da2-e2fe-4817-9b4f-4cd05e5e93cb None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:12.254615 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6979da2-e2fe-4817-9b4f-4cd05e5e93cb None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:12.266449 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6979da2-e2fe-4817-9b4f-4cd05e5e93cb None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:12.268078 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6979da2-e2fe-4817-9b4f-4cd05e5e93cb None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:12.287083 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application [None req-e6979da2-e2fe-4817-9b4f-4cd05e5e93cb None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:54:12.287083 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:54:12.287083 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:54:12.287083 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:54:12.287083 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:54:12.287083 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:54:12.287083 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:54:12.287083 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:54:12.287083 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:54:12.287083 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:54:12.287083 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:54:12.287083 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:54:12.287083 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 13 15:54:12.287083 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 13 15:54:12.287083 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 13 15:54:12.287083 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 13 15:54:12.288081 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:54:12.288081 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:54:12.288081 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:54:12.288081 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:54:12.288081 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:54:12.288081 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:54:12.288081 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:54:12.288081 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application with Lock( Sep 13 15:54:12.288081 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:54:12.288081 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:54:12.288081 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:54:12.288081 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:54:12.288081 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:54:12.288081 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:54:12.288081 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:54:12.288081 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:54:12.288081 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 13 15:54:12.288961 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 13 15:54:12.288961 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:54:12.288961 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Sep 13 15:54:12.296198 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 298/600] 10.222.0.50 () {56 vars in 1108 bytes} [Tue Sep 13 15:54:12 2022] GET /identity/v3/domains/Default => generated 87 bytes in 199 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:54:12.296198 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6979da2-e2fe-4817-9b4f-4cd05e5e93cb None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:12.341776 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72b6fec9-3740-4e0d-b33c-dbbb4bc10ec1 None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:12.379279 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c239f2d7-3466-4767-9dab-e81748e804bf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:12.379804 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c239f2d7-3466-4767-9dab-e81748e804bf None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:12.380153 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c239f2d7-3466-4767-9dab-e81748e804bf None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:12.380349 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c239f2d7-3466-4767-9dab-e81748e804bf None admin] PATH_INFO: `/v3/projects` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:12.381033 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c239f2d7-3466-4767-9dab-e81748e804bf None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81437) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:54:12.381337 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c239f2d7-3466-4767-9dab-e81748e804bf None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:12.383188 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c239f2d7-3466-4767-9dab-e81748e804bf None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:12.401777 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 303/601] 10.222.0.50 () {56 vars in 1148 bytes} [Tue Sep 13 15:54:12 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 193 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:54:12.401777 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c239f2d7-3466-4767-9dab-e81748e804bf None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:12.417631 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:54:12.440990 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6979da2-e2fe-4817-9b4f-4cd05e5e93cb None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:12.456047 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:54:12.484927 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9887005-4216-4fac-afe1-980fba1f256b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:12.486437 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9887005-4216-4fac-afe1-980fba1f256b None admin] REQUEST_METHOD: `PUT` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:12.486951 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9887005-4216-4fac-afe1-980fba1f256b None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:12.487488 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9887005-4216-4fac-afe1-980fba1f256b None admin] PATH_INFO: `/v3/projects/5811e3b83a1e462e9320cad79a3ad791/users/7fd6b5afd67b48f9b0f0597ccd4cf4df/roles/da847124d9374ac88b27968451c486ab` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:12.504650 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9887005-4216-4fac-afe1-980fba1f256b None admin] RBAC: Authorizing `identity:create_grant(project_id=5811e3b83a1e462e9320cad79a3ad791, user_id=7fd6b5afd67b48f9b0f0597ccd4cf4df, role_id=da847124d9374ac88b27968451c486ab)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:12.506207 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9887005-4216-4fac-afe1-980fba1f256b None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:12.515641 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c239f2d7-3466-4767-9dab-e81748e804bf None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:12.527098 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 299/602] 10.222.0.50 () {58 vars in 1439 bytes} [Tue Sep 13 15:54:12 2022] PUT /identity/v3/projects/5811e3b83a1e462e9320cad79a3ad791/users/7fd6b5afd67b48f9b0f0597ccd4cf4df/roles/da847124d9374ac88b27968451c486ab => generated 0 bytes in 226 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:54:12.528056 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9887005-4216-4fac-afe1-980fba1f256b None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:12.559040 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4312e26-42cb-4c7d-a1bc-ff6b2828f55f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:12.560141 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4312e26-42cb-4c7d-a1bc-ff6b2828f55f None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:12.560630 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4312e26-42cb-4c7d-a1bc-ff6b2828f55f None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:12.561146 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4312e26-42cb-4c7d-a1bc-ff6b2828f55f None admin] PATH_INFO: `/v3/domains` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:12.562140 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4312e26-42cb-4c7d-a1bc-ff6b2828f55f None admin] RBAC: Adding query filter params (name=Default) {{(pid=81437) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:54:12.562764 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4312e26-42cb-4c7d-a1bc-ff6b2828f55f None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:12.564563 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4312e26-42cb-4c7d-a1bc-ff6b2828f55f None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:12.579558 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 304/603] 10.222.0.50 () {56 vars in 1109 bytes} [Tue Sep 13 15:54:12 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:54:12.585692 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4312e26-42cb-4c7d-a1bc-ff6b2828f55f None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:12.659587 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9887005-4216-4fac-afe1-980fba1f256b None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:12.700948 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f91b270a-0640-4421-9321-14145034026e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:12.701636 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f91b270a-0640-4421-9321-14145034026e None admin] REQUEST_METHOD: `PUT` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:12.701918 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f91b270a-0640-4421-9321-14145034026e None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:12.702193 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f91b270a-0640-4421-9321-14145034026e None admin] PATH_INFO: `/v3/projects/5811e3b83a1e462e9320cad79a3ad791/users/6ab5f8018f464be69d4af7683f553738/roles/da847124d9374ac88b27968451c486ab` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:12.715326 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:54:12.723361 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4312e26-42cb-4c7d-a1bc-ff6b2828f55f None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:12.724019 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f91b270a-0640-4421-9321-14145034026e None admin] RBAC: Authorizing `identity:create_grant(project_id=5811e3b83a1e462e9320cad79a3ad791, user_id=6ab5f8018f464be69d4af7683f553738, role_id=da847124d9374ac88b27968451c486ab)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:12.725577 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f91b270a-0640-4421-9321-14145034026e None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:12.736964 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:54:12.747942 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 300/604] 10.222.0.50 () {58 vars in 1439 bytes} [Tue Sep 13 15:54:12 2022] PUT /identity/v3/projects/5811e3b83a1e462e9320cad79a3ad791/users/6ab5f8018f464be69d4af7683f553738/roles/da847124d9374ac88b27968451c486ab => generated 0 bytes in 220 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:54:12.752547 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b9c016b-269f-4fc3-a663-04873ba56aa5 None None] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:12.752547 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b9c016b-269f-4fc3-a663-04873ba56aa5 None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:12.752547 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b9c016b-269f-4fc3-a663-04873ba56aa5 None None] PATH_INFO: `/` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:12.752547 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 301/605] 10.222.0.50 () {54 vars in 927 bytes} [Tue Sep 13 15:54:12 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:54:12.756997 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-497155bb-460e-4937-86f4-f05a8d62841f None None] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:12.757284 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-497155bb-460e-4937-86f4-f05a8d62841f None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:12.757559 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-497155bb-460e-4937-86f4-f05a8d62841f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:12.778161 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.auth.core [None req-497155bb-460e-4937-86f4-f05a8d62841f None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81438) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:54:12.780092 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-497155bb-460e-4937-86f4-f05a8d62841f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:54:12.783538 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed97fd6c-96f6-42e9-851d-6baa9641b0a4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:12.784592 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed97fd6c-96f6-42e9-851d-6baa9641b0a4 None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:12.785093 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed97fd6c-96f6-42e9-851d-6baa9641b0a4 None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:12.785658 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed97fd6c-96f6-42e9-851d-6baa9641b0a4 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:12.794981 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed97fd6c-96f6-42e9-851d-6baa9641b0a4 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:12.796887 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed97fd6c-96f6-42e9-851d-6baa9641b0a4 None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:12.811693 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application [None req-ed97fd6c-96f6-42e9-851d-6baa9641b0a4 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 13 15:54:12.811693 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:54:12.811693 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:54:12.811693 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:54:12.811693 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:54:12.811693 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:54:12.811693 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:54:12.811693 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:54:12.811693 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:54:12.811693 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:54:12.811693 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:54:12.811693 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:54:12.811693 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 13 15:54:12.811693 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 13 15:54:12.811693 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 13 15:54:12.812875 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 13 15:54:12.812875 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:54:12.812875 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:54:12.812875 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:54:12.812875 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:54:12.812875 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:54:12.812875 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:54:12.812875 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:54:12.812875 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application with Lock( Sep 13 15:54:12.812875 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:54:12.812875 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:54:12.812875 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:54:12.812875 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:54:12.812875 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:54:12.812875 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:54:12.812875 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:54:12.812875 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:54:12.813814 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 13 15:54:12.813814 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 13 15:54:12.813814 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 13 15:54:12.813814 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 13 15:54:12.813814 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 13 15:54:12.813814 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 13 15:54:12.813814 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 13 15:54:12.813814 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application Sep 13 15:54:12.816402 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 305/606] 10.222.0.50 () {56 vars in 1111 bytes} [Tue Sep 13 15:54:12 2022] GET /identity/v3/projects/service => generated 88 bytes in 231 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 13 15:54:12.821944 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed97fd6c-96f6-42e9-851d-6baa9641b0a4 None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:12.870014 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 302/607] 10.222.0.50 () {58 vars in 1024 bytes} [Tue Sep 13 15:54:12 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 113 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 13 15:54:12.964465 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed97fd6c-96f6-42e9-851d-6baa9641b0a4 None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:13.003377 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17cb5b9b-c900-4109-ab7d-86906b82bf10 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:13.003982 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17cb5b9b-c900-4109-ab7d-86906b82bf10 None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:13.004260 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17cb5b9b-c900-4109-ab7d-86906b82bf10 None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:13.004534 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17cb5b9b-c900-4109-ab7d-86906b82bf10 None admin] PATH_INFO: `/v3/projects` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:13.005268 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17cb5b9b-c900-4109-ab7d-86906b82bf10 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81437) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:54:13.005580 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17cb5b9b-c900-4109-ab7d-86906b82bf10 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:13.007031 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17cb5b9b-c900-4109-ab7d-86906b82bf10 None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:13.020977 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 306/608] 10.222.0.50 () {56 vars in 1148 bytes} [Tue Sep 13 15:54:12 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 199 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 13 15:54:13.026761 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-497155bb-460e-4937-86f4-f05a8d62841f None None] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:13.156545 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-497155bb-460e-4937-86f4-f05a8d62841f None None] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:13.181990 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:54:13.184269 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17cb5b9b-c900-4109-ab7d-86906b82bf10 None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:13.209624 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40dec70e-81c8-4541-ae10-3bedb5a5d381 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:13.210288 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40dec70e-81c8-4541-ae10-3bedb5a5d381 None admin] REQUEST_METHOD: `PUT` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:13.211049 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40dec70e-81c8-4541-ae10-3bedb5a5d381 None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:13.211332 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40dec70e-81c8-4541-ae10-3bedb5a5d381 None admin] PATH_INFO: `/v3/projects/5811e3b83a1e462e9320cad79a3ad791/users/2f436f07939d4855a21b195bc7566244/roles/da847124d9374ac88b27968451c486ab` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:13.227418 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40dec70e-81c8-4541-ae10-3bedb5a5d381 None admin] RBAC: Authorizing `identity:create_grant(project_id=5811e3b83a1e462e9320cad79a3ad791, user_id=2f436f07939d4855a21b195bc7566244, role_id=da847124d9374ac88b27968451c486ab)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:13.230024 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40dec70e-81c8-4541-ae10-3bedb5a5d381 None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:13.242105 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:54:13.251702 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 303/609] 10.222.0.50 () {58 vars in 1439 bytes} [Tue Sep 13 15:54:13 2022] PUT /identity/v3/projects/5811e3b83a1e462e9320cad79a3ad791/users/2f436f07939d4855a21b195bc7566244/roles/da847124d9374ac88b27968451c486ab => generated 0 bytes in 225 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:54:13.337298 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17cb5b9b-c900-4109-ab7d-86906b82bf10 None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:13.386269 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55b6244e-94ea-4521-800b-7c0f6bb4ae80 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:13.386945 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55b6244e-94ea-4521-800b-7c0f6bb4ae80 None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:13.387203 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55b6244e-94ea-4521-800b-7c0f6bb4ae80 None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:13.387573 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55b6244e-94ea-4521-800b-7c0f6bb4ae80 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:13.401646 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55b6244e-94ea-4521-800b-7c0f6bb4ae80 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 13 15:54:13.406573 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application [None req-55b6244e-94ea-4521-800b-7c0f6bb4ae80 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 13 15:54:13.406573 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:54:13.406573 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:54:13.406573 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:54:13.406573 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:54:13.406573 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:54:13.406573 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:54:13.406573 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:54:13.406573 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:54:13.406573 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:54:13.406573 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:54:13.406573 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:54:13.406573 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 13 15:54:13.406573 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 13 15:54:13.406573 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 13 15:54:13.406573 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 13 15:54:13.407550 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 13 15:54:13.407550 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 13 15:54:13.407550 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 13 15:54:13.407550 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 13 15:54:13.407550 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:54:13.407550 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:54:13.407550 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:54:13.407550 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:54:13.407550 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:54:13.407550 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:54:13.407550 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:54:13.407550 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application with Lock( Sep 13 15:54:13.407550 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:54:13.407550 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:54:13.407550 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:54:13.407550 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:54:13.408977 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:54:13.408977 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:54:13.408977 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:54:13.408977 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:54:13.408977 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 13 15:54:13.408977 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 13 15:54:13.408977 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 13 15:54:13.408977 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 13 15:54:13.408977 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 13 15:54:13.408977 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 13 15:54:13.408977 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 13 15:54:13.408977 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application Sep 13 15:54:13.408977 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 307/610] 10.222.0.50 () {56 vars in 1102 bytes} [Tue Sep 13 15:54:13 2022] GET /identity/v3/roles/service => generated 85 bytes in 224 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:54:13.413815 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40dec70e-81c8-4541-ae10-3bedb5a5d381 None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:13.562167 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40dec70e-81c8-4541-ae10-3bedb5a5d381 None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:13.601262 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac1373ee-fc56-4302-a02c-8bd685ae6446 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:13.601873 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac1373ee-fc56-4302-a02c-8bd685ae6446 None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:13.602147 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac1373ee-fc56-4302-a02c-8bd685ae6446 None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:13.602413 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac1373ee-fc56-4302-a02c-8bd685ae6446 None admin] PATH_INFO: `/v3/roles` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:13.603647 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac1373ee-fc56-4302-a02c-8bd685ae6446 None admin] RBAC: Adding query filter params (name=service) {{(pid=81438) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:54:13.604016 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac1373ee-fc56-4302-a02c-8bd685ae6446 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:13.605382 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac1373ee-fc56-4302-a02c-8bd685ae6446 None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:13.614798 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 304/611] 10.222.0.50 () {56 vars in 1103 bytes} [Tue Sep 13 15:54:13 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 201 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:54:13.624277 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55b6244e-94ea-4521-800b-7c0f6bb4ae80 None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:13.747289 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:54:13.765037 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55b6244e-94ea-4521-800b-7c0f6bb4ae80 None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:13.812101 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7b3f744-66e3-403c-8f6a-7e4022bc0c09 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:13.812793 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7b3f744-66e3-403c-8f6a-7e4022bc0c09 None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:13.813013 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7b3f744-66e3-403c-8f6a-7e4022bc0c09 None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:13.813291 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7b3f744-66e3-403c-8f6a-7e4022bc0c09 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:13.815259 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1d2ebd1-4a22-4a36-aa59-ff2c7dd3b4fd None None] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:13.815556 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1d2ebd1-4a22-4a36-aa59-ff2c7dd3b4fd None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:13.815841 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1d2ebd1-4a22-4a36-aa59-ff2c7dd3b4fd None None] PATH_INFO: `/` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:13.817445 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 305/612] 10.222.0.50 () {54 vars in 927 bytes} [Tue Sep 13 15:54:13 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:54:13.822986 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2dea8f8-d4a1-40d3-bb97-546d631cd3ac None None] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:13.823275 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2dea8f8-d4a1-40d3-bb97-546d631cd3ac None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:13.823559 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2dea8f8-d4a1-40d3-bb97-546d631cd3ac None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:13.829192 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7b3f744-66e3-403c-8f6a-7e4022bc0c09 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:13.831374 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7b3f744-66e3-403c-8f6a-7e4022bc0c09 None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:13.853091 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.auth.core [None req-d2dea8f8-d4a1-40d3-bb97-546d631cd3ac None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81438) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:54:13.854945 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-d2dea8f8-d4a1-40d3-bb97-546d631cd3ac None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:54:13.862533 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application [None req-c7b3f744-66e3-403c-8f6a-7e4022bc0c09 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:54:13.862533 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:54:13.862533 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:54:13.862533 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:54:13.862533 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:54:13.862533 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:54:13.862533 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:54:13.862533 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:54:13.862533 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:54:13.862533 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:54:13.862533 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:54:13.862533 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:54:13.862533 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 13 15:54:13.862533 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 13 15:54:13.862533 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 13 15:54:13.862533 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 13 15:54:13.863579 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:54:13.863579 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:54:13.863579 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:54:13.863579 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:54:13.863579 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:54:13.863579 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:54:13.863579 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:54:13.863579 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application with Lock( Sep 13 15:54:13.863579 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:54:13.863579 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:54:13.863579 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:54:13.863579 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:54:13.863579 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:54:13.863579 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:54:13.863579 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:54:13.863579 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:54:13.863579 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 13 15:54:13.865252 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 13 15:54:13.865252 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:54:13.865252 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application Sep 13 15:54:13.865252 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 308/613] 10.222.0.50 () {56 vars in 1108 bytes} [Tue Sep 13 15:54:13 2022] GET /identity/v3/domains/Default => generated 87 bytes in 240 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:54:13.869488 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7b3f744-66e3-403c-8f6a-7e4022bc0c09 None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:13.892134 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:54:13.976558 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 306/614] 10.222.0.50 () {58 vars in 1024 bytes} [Tue Sep 13 15:54:13 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 154 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 13 15:54:14.023617 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7b3f744-66e3-403c-8f6a-7e4022bc0c09 None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:14.060672 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2137ab4-1c8d-4268-9af0-c5750f84023d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:14.061293 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2137ab4-1c8d-4268-9af0-c5750f84023d None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:14.061574 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2137ab4-1c8d-4268-9af0-c5750f84023d None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:14.061848 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2137ab4-1c8d-4268-9af0-c5750f84023d None admin] PATH_INFO: `/v3/domains` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:14.062498 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2137ab4-1c8d-4268-9af0-c5750f84023d None admin] RBAC: Adding query filter params (name=Default) {{(pid=81437) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:54:14.062857 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2137ab4-1c8d-4268-9af0-c5750f84023d None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:14.064226 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2137ab4-1c8d-4268-9af0-c5750f84023d None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:14.075255 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:54:14.077599 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 309/615] 10.222.0.50 () {56 vars in 1109 bytes} [Tue Sep 13 15:54:13 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 208 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:54:14.083834 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2dea8f8-d4a1-40d3-bb97-546d631cd3ac None None] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:14.212139 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2dea8f8-d4a1-40d3-bb97-546d631cd3ac None None] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:14.257268 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fb5cda3-8d65-4a32-8971-ee9d1ecf919f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:14.257906 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fb5cda3-8d65-4a32-8971-ee9d1ecf919f None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:14.258187 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fb5cda3-8d65-4a32-8971-ee9d1ecf919f None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:14.258463 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fb5cda3-8d65-4a32-8971-ee9d1ecf919f None admin] PATH_INFO: `/v3/users/swift` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:14.266649 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fb5cda3-8d65-4a32-8971-ee9d1ecf919f None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:14.268145 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fb5cda3-8d65-4a32-8971-ee9d1ecf919f None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:14.284571 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application [None req-1fb5cda3-8d65-4a32-8971-ee9d1ecf919f None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Sep 13 15:54:14.284571 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:54:14.284571 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:54:14.284571 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:54:14.284571 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:54:14.284571 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:54:14.284571 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:54:14.284571 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:54:14.284571 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:54:14.284571 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:54:14.284571 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:54:14.284571 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:54:14.284571 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 13 15:54:14.284571 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 13 15:54:14.284571 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 13 15:54:14.284571 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 13 15:54:14.285585 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:54:14.285585 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:54:14.285585 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 13 15:54:14.285585 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 13 15:54:14.285585 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 13 15:54:14.285585 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 13 15:54:14.285585 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:54:14.285585 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:54:14.285585 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:54:14.285585 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:54:14.285585 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:54:14.285585 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application with Lock( Sep 13 15:54:14.285585 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:54:14.285585 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:54:14.285585 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:54:14.285585 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:54:14.285585 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:54:14.285585 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:54:14.287190 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:54:14.287190 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:54:14.287190 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 13 15:54:14.287190 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 13 15:54:14.287190 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 13 15:54:14.287190 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 13 15:54:14.287190 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 13 15:54:14.287190 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 13 15:54:14.287190 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Sep 13 15:54:14.287190 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Sep 13 15:54:14.287190 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 307/616] 10.222.0.50 () {56 vars in 1096 bytes} [Tue Sep 13 15:54:14 2022] GET /identity/v3/users/swift => generated 83 bytes in 203 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:54:14.291722 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2137ab4-1c8d-4268-9af0-c5750f84023d None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:14.311279 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fb5cda3-8d65-4a32-8971-ee9d1ecf919f None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:14.336809 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:54:14.468358 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2137ab4-1c8d-4268-9af0-c5750f84023d None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:14.489436 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fb5cda3-8d65-4a32-8971-ee9d1ecf919f None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:14.539214 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-177677d8-bf41-4b5a-8b23-71c24de05f7e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:14.539829 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-177677d8-bf41-4b5a-8b23-71c24de05f7e None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:14.540107 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-177677d8-bf41-4b5a-8b23-71c24de05f7e None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:14.540379 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-177677d8-bf41-4b5a-8b23-71c24de05f7e None admin] PATH_INFO: `/v3/users` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:14.541555 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-177677d8-bf41-4b5a-8b23-71c24de05f7e None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=81437) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:54:14.541871 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-177677d8-bf41-4b5a-8b23-71c24de05f7e None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:14.543346 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-177677d8-bf41-4b5a-8b23-71c24de05f7e None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:14.549571 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0910acb5-d239-4dec-8876-16c540a6d204 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:14.550217 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0910acb5-d239-4dec-8876-16c540a6d204 None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:14.550499 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0910acb5-d239-4dec-8876-16c540a6d204 None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:14.550774 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0910acb5-d239-4dec-8876-16c540a6d204 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:14.574180 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0910acb5-d239-4dec-8876-16c540a6d204 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 13 15:54:14.580634 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application [None req-0910acb5-d239-4dec-8876-16c540a6d204 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 13 15:54:14.580634 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:54:14.580634 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:54:14.580634 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:54:14.580634 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:54:14.580634 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:54:14.580634 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:54:14.580634 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:54:14.580634 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:54:14.580634 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:54:14.580634 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:54:14.580634 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:54:14.580634 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 13 15:54:14.580634 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 13 15:54:14.580634 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 13 15:54:14.580634 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 13 15:54:14.581571 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 13 15:54:14.581571 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 13 15:54:14.581571 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 13 15:54:14.581571 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 13 15:54:14.581571 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:54:14.581571 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:54:14.581571 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:54:14.581571 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:54:14.581571 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:54:14.581571 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:54:14.581571 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:54:14.581571 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application with Lock( Sep 13 15:54:14.581571 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:54:14.581571 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:54:14.581571 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:54:14.581571 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:54:14.582432 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:54:14.582432 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:54:14.582432 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:54:14.582432 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:54:14.582432 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 13 15:54:14.582432 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 13 15:54:14.582432 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 13 15:54:14.582432 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 13 15:54:14.582432 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 13 15:54:14.582432 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 13 15:54:14.582432 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 13 15:54:14.582432 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Sep 13 15:54:14.582432 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 308/617] 10.222.0.50 () {56 vars in 1102 bytes} [Tue Sep 13 15:54:14 2022] GET /identity/v3/roles/service => generated 85 bytes in 271 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 13 15:54:14.584104 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 310/618] 10.222.0.50 () {56 vars in 1135 bytes} [Tue Sep 13 15:54:14 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 293 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:54:14.589996 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0910acb5-d239-4dec-8876-16c540a6d204 None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:14.591229 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-177677d8-bf41-4b5a-8b23-71c24de05f7e None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:14.682953 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:54:14.720956 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:54:14.734224 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-177677d8-bf41-4b5a-8b23-71c24de05f7e None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:14.767901 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0910acb5-d239-4dec-8876-16c540a6d204 None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:14.797787 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e59bbdc-526e-4fca-8408-c9c3821f2e00 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:14.798447 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e59bbdc-526e-4fca-8408-c9c3821f2e00 None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:14.798676 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e59bbdc-526e-4fca-8408-c9c3821f2e00 None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:14.798981 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e59bbdc-526e-4fca-8408-c9c3821f2e00 None admin] PATH_INFO: `/v3/roles` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:14.799716 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e59bbdc-526e-4fca-8408-c9c3821f2e00 None admin] RBAC: Adding query filter params (name=service) {{(pid=81437) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:54:14.800027 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e59bbdc-526e-4fca-8408-c9c3821f2e00 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:14.801412 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e59bbdc-526e-4fca-8408-c9c3821f2e00 None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:14.810384 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 311/619] 10.222.0.50 () {56 vars in 1103 bytes} [Tue Sep 13 15:54:14 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 219 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:54:14.817740 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e59bbdc-526e-4fca-8408-c9c3821f2e00 None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:14.823407 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5524c0e5-d310-4678-aba5-1f4a1397dcd9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:14.824058 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5524c0e5-d310-4678-aba5-1f4a1397dcd9 None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:14.824343 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5524c0e5-d310-4678-aba5-1f4a1397dcd9 None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:14.824621 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5524c0e5-d310-4678-aba5-1f4a1397dcd9 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:14.835644 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5524c0e5-d310-4678-aba5-1f4a1397dcd9 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:14.838147 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5524c0e5-d310-4678-aba5-1f4a1397dcd9 None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:14.853409 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application [None req-5524c0e5-d310-4678-aba5-1f4a1397dcd9 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:54:14.853409 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:54:14.853409 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:54:14.853409 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:54:14.853409 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:54:14.853409 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:54:14.853409 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:54:14.853409 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:54:14.853409 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:54:14.853409 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:54:14.853409 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:54:14.853409 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:54:14.853409 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 13 15:54:14.853409 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 13 15:54:14.853409 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 13 15:54:14.853409 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 13 15:54:14.854405 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:54:14.854405 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:54:14.854405 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:54:14.854405 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:54:14.854405 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:54:14.854405 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:54:14.854405 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:54:14.854405 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application with Lock( Sep 13 15:54:14.854405 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:54:14.854405 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:54:14.854405 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:54:14.854405 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:54:14.854405 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:54:14.854405 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:54:14.854405 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:54:14.854405 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:54:14.854405 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 13 15:54:14.855345 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 13 15:54:14.855345 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:54:14.855345 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Sep 13 15:54:14.855345 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 309/620] 10.222.0.50 () {56 vars in 1108 bytes} [Tue Sep 13 15:54:14 2022] GET /identity/v3/domains/Default => generated 87 bytes in 265 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 13 15:54:14.863947 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5524c0e5-d310-4678-aba5-1f4a1397dcd9 None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:14.993361 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e59bbdc-526e-4fca-8408-c9c3821f2e00 None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:15.031286 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:54:15.054614 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5524c0e5-d310-4678-aba5-1f4a1397dcd9 None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:15.061658 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f31dcf9-3e71-45b1-a3d4-5e66c539ee48 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:15.062383 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f31dcf9-3e71-45b1-a3d4-5e66c539ee48 None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:15.062669 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f31dcf9-3e71-45b1-a3d4-5e66c539ee48 None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:15.062966 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f31dcf9-3e71-45b1-a3d4-5e66c539ee48 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:15.081442 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f31dcf9-3e71-45b1-a3d4-5e66c539ee48 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:15.083033 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f31dcf9-3e71-45b1-a3d4-5e66c539ee48 None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:15.101528 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application [None req-6f31dcf9-3e71-45b1-a3d4-5e66c539ee48 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:54:15.101528 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:54:15.101528 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:54:15.101528 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:54:15.101528 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:54:15.101528 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:54:15.101528 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:54:15.101528 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:54:15.101528 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:54:15.101528 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:54:15.101528 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:54:15.101528 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:54:15.101528 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 13 15:54:15.101528 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 13 15:54:15.101528 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 13 15:54:15.101528 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 13 15:54:15.102547 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:54:15.102547 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:54:15.102547 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:54:15.102547 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:54:15.102547 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:54:15.102547 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:54:15.102547 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:54:15.102547 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application with Lock( Sep 13 15:54:15.102547 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:54:15.102547 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:54:15.102547 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:54:15.102547 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:54:15.102547 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:54:15.102547 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:54:15.102547 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:54:15.102547 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:54:15.102547 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 13 15:54:15.104096 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 13 15:54:15.104096 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:54:15.104096 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application Sep 13 15:54:15.110926 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 312/621] 10.222.0.50 () {56 vars in 1108 bytes} [Tue Sep 13 15:54:14 2022] GET /identity/v3/domains/Default => generated 87 bytes in 288 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:54:15.112366 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f31dcf9-3e71-45b1-a3d4-5e66c539ee48 None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:15.114610 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:54:15.125365 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c3938f1-b74d-41fb-949e-902e108e30bb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:15.126154 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c3938f1-b74d-41fb-949e-902e108e30bb None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:15.126538 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c3938f1-b74d-41fb-949e-902e108e30bb None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:15.126947 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c3938f1-b74d-41fb-949e-902e108e30bb None admin] PATH_INFO: `/v3/domains` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:15.127703 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c3938f1-b74d-41fb-949e-902e108e30bb None admin] RBAC: Adding query filter params (name=Default) {{(pid=81438) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:54:15.128112 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c3938f1-b74d-41fb-949e-902e108e30bb None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:15.131244 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c3938f1-b74d-41fb-949e-902e108e30bb None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:15.159561 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 310/622] 10.222.0.50 () {56 vars in 1109 bytes} [Tue Sep 13 15:54:14 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 295 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:54:15.161538 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c3938f1-b74d-41fb-949e-902e108e30bb None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:15.308992 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c3938f1-b74d-41fb-949e-902e108e30bb None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:15.356393 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e4f0570-c5c5-48e0-ae3f-f1fc29dad92d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:15.357290 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e4f0570-c5c5-48e0-ae3f-f1fc29dad92d None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:15.357728 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e4f0570-c5c5-48e0-ae3f-f1fc29dad92d None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:15.358135 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e4f0570-c5c5-48e0-ae3f-f1fc29dad92d None admin] PATH_INFO: `/v3/projects/service` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:15.366110 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e4f0570-c5c5-48e0-ae3f-f1fc29dad92d None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:15.367823 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e4f0570-c5c5-48e0-ae3f-f1fc29dad92d None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:15.382841 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f31dcf9-3e71-45b1-a3d4-5e66c539ee48 None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:15.383508 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application [None req-2e4f0570-c5c5-48e0-ae3f-f1fc29dad92d None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 13 15:54:15.383508 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:54:15.383508 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:54:15.383508 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:54:15.383508 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:54:15.383508 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:54:15.383508 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:54:15.383508 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:54:15.383508 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:54:15.383508 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:54:15.383508 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:54:15.383508 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:54:15.383508 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 13 15:54:15.383508 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 13 15:54:15.383508 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 13 15:54:15.385142 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 13 15:54:15.385142 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:54:15.385142 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:54:15.385142 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:54:15.385142 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:54:15.385142 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:54:15.385142 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:54:15.385142 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:54:15.385142 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application with Lock( Sep 13 15:54:15.385142 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:54:15.385142 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:54:15.385142 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:54:15.385142 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:54:15.385142 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:54:15.385142 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:54:15.385142 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:54:15.385142 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:54:15.386126 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 13 15:54:15.386126 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 13 15:54:15.386126 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 13 15:54:15.386126 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 13 15:54:15.386126 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 13 15:54:15.386126 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 13 15:54:15.386126 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 13 15:54:15.386126 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Sep 13 15:54:15.388362 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 311/623] 10.222.0.50 () {56 vars in 1111 bytes} [Tue Sep 13 15:54:15 2022] GET /identity/v3/projects/service => generated 88 bytes in 227 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:54:15.395921 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e4f0570-c5c5-48e0-ae3f-f1fc29dad92d None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:15.429133 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4414c1f4-5df7-4e97-9214-d8497d8e4946 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:15.430000 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4414c1f4-5df7-4e97-9214-d8497d8e4946 None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:15.430432 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4414c1f4-5df7-4e97-9214-d8497d8e4946 None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:15.430878 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4414c1f4-5df7-4e97-9214-d8497d8e4946 None admin] PATH_INFO: `/v3/domains` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:15.432119 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4414c1f4-5df7-4e97-9214-d8497d8e4946 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81437) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:54:15.433098 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4414c1f4-5df7-4e97-9214-d8497d8e4946 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:15.435158 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4414c1f4-5df7-4e97-9214-d8497d8e4946 None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:15.450432 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 313/624] 10.222.0.50 () {56 vars in 1109 bytes} [Tue Sep 13 15:54:15 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 339 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 13 15:54:15.461177 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4414c1f4-5df7-4e97-9214-d8497d8e4946 None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:15.495597 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:54:15.506617 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:54:15.568488 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e4f0570-c5c5-48e0-ae3f-f1fc29dad92d None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:15.601887 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4414c1f4-5df7-4e97-9214-d8497d8e4946 None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:15.615238 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-928a5183-12d3-4d21-9508-4ac367a1fd58 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:15.616108 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-928a5183-12d3-4d21-9508-4ac367a1fd58 None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:15.616298 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-928a5183-12d3-4d21-9508-4ac367a1fd58 None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:15.616674 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-928a5183-12d3-4d21-9508-4ac367a1fd58 None admin] PATH_INFO: `/v3/projects` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:15.617512 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-928a5183-12d3-4d21-9508-4ac367a1fd58 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81438) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:54:15.617891 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-928a5183-12d3-4d21-9508-4ac367a1fd58 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:15.619844 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-928a5183-12d3-4d21-9508-4ac367a1fd58 None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:15.632315 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 312/625] 10.222.0.50 () {56 vars in 1148 bytes} [Tue Sep 13 15:54:15 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 237 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:54:15.638266 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4750360-5fca-4f90-97ac-7225be80f633 None None] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:15.638266 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4750360-5fca-4f90-97ac-7225be80f633 None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:15.638266 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4750360-5fca-4f90-97ac-7225be80f633 None None] PATH_INFO: `/` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:15.638266 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 313/626] 10.222.0.50 () {54 vars in 927 bytes} [Tue Sep 13 15:54:15 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:54:15.638266 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9276deb-436d-43a0-9b7e-f5c12fc2ad0b None None] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:15.638266 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9276deb-436d-43a0-9b7e-f5c12fc2ad0b None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:15.638751 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9276deb-436d-43a0-9b7e-f5c12fc2ad0b None None] PATH_INFO: `/` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:15.638751 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 314/627] 10.222.0.50 () {54 vars in 927 bytes} [Tue Sep 13 15:54:15 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:54:15.644171 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9276deb-436d-43a0-9b7e-f5c12fc2ad0b None None] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:15.656770 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a213b0b-b967-494c-b033-4b4ac047aa7e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:15.657325 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a213b0b-b967-494c-b033-4b4ac047aa7e None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:15.657464 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a213b0b-b967-494c-b033-4b4ac047aa7e None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:15.657755 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a213b0b-b967-494c-b033-4b4ac047aa7e None admin] PATH_INFO: `/v3/users/nova` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:15.665439 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a213b0b-b967-494c-b033-4b4ac047aa7e None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:15.667239 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a213b0b-b967-494c-b033-4b4ac047aa7e None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:15.680956 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application [None req-3a213b0b-b967-494c-b033-4b4ac047aa7e None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Sep 13 15:54:15.680956 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:54:15.680956 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:54:15.680956 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:54:15.680956 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:54:15.680956 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:54:15.680956 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:54:15.680956 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:54:15.680956 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:54:15.680956 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:54:15.680956 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:54:15.680956 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:54:15.680956 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 13 15:54:15.680956 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 13 15:54:15.680956 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 13 15:54:15.680956 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 13 15:54:15.682583 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:54:15.682583 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:54:15.682583 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 13 15:54:15.682583 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 13 15:54:15.682583 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 13 15:54:15.682583 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 13 15:54:15.682583 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:54:15.682583 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:54:15.682583 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:54:15.682583 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:54:15.682583 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:54:15.682583 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application with Lock( Sep 13 15:54:15.682583 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:54:15.682583 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:54:15.682583 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:54:15.682583 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:54:15.682583 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:54:15.682583 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:54:15.683836 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:54:15.683836 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:54:15.683836 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 13 15:54:15.683836 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 13 15:54:15.683836 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 13 15:54:15.683836 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 13 15:54:15.683836 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 13 15:54:15.683836 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 13 15:54:15.683836 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Sep 13 15:54:15.683836 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application Sep 13 15:54:15.683836 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 314/628] 10.222.0.50 () {56 vars in 1093 bytes} [Tue Sep 13 15:54:15 2022] GET /identity/v3/users/nova => generated 82 bytes in 223 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:54:15.684506 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09d65e34-f087-4115-aa57-73b0766994be None None] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:15.684506 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09d65e34-f087-4115-aa57-73b0766994be None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:15.684649 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09d65e34-f087-4115-aa57-73b0766994be None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:15.705257 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.auth.core [None req-09d65e34-f087-4115-aa57-73b0766994be None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81437) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:54:15.706439 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-09d65e34-f087-4115-aa57-73b0766994be None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:54:15.780258 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9276deb-436d-43a0-9b7e-f5c12fc2ad0b None None] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:15.791160 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:54:15.798842 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 315/629] 10.222.0.50 () {58 vars in 1024 bytes} [Tue Sep 13 15:54:15 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 13 15:54:15.798842 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67b794f2-e6ab-476d-825c-bafdf2e4f36b None None] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:15.798842 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67b794f2-e6ab-476d-825c-bafdf2e4f36b None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:15.798842 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67b794f2-e6ab-476d-825c-bafdf2e4f36b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:15.819969 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.auth.core [None req-67b794f2-e6ab-476d-825c-bafdf2e4f36b None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81437) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:54:15.822328 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-67b794f2-e6ab-476d-825c-bafdf2e4f36b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:54:15.826715 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de5647fd-4d04-42b2-8524-da3b49bfc5ad None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:15.831293 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de5647fd-4d04-42b2-8524-da3b49bfc5ad None admin] REQUEST_METHOD: `PUT` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:15.831293 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de5647fd-4d04-42b2-8524-da3b49bfc5ad None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:15.831293 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de5647fd-4d04-42b2-8524-da3b49bfc5ad None admin] PATH_INFO: `/v3/projects/5811e3b83a1e462e9320cad79a3ad791/users/b00ab0d6321d483188bc69deaddcdbbc/roles/da847124d9374ac88b27968451c486ab` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:15.847947 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:54:15.850136 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de5647fd-4d04-42b2-8524-da3b49bfc5ad None admin] RBAC: Authorizing `identity:create_grant(project_id=5811e3b83a1e462e9320cad79a3ad791, user_id=b00ab0d6321d483188bc69deaddcdbbc, role_id=da847124d9374ac88b27968451c486ab)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:15.851698 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de5647fd-4d04-42b2-8524-da3b49bfc5ad None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:15.871070 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 315/630] 10.222.0.50 () {58 vars in 1439 bytes} [Tue Sep 13 15:54:15 2022] PUT /identity/v3/projects/5811e3b83a1e462e9320cad79a3ad791/users/b00ab0d6321d483188bc69deaddcdbbc/roles/da847124d9374ac88b27968451c486ab => generated 0 bytes in 224 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:54:15.872430 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de5647fd-4d04-42b2-8524-da3b49bfc5ad None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:15.959598 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 316/631] 10.222.0.50 () {58 vars in 1024 bytes} [Tue Sep 13 15:54:15 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 166 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 13 15:54:16.033861 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de5647fd-4d04-42b2-8524-da3b49bfc5ad None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:16.070793 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f99e849-c002-4230-b470-d15d67b4696b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:16.071449 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f99e849-c002-4230-b470-d15d67b4696b None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:16.072410 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f99e849-c002-4230-b470-d15d67b4696b None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:16.073162 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f99e849-c002-4230-b470-d15d67b4696b None admin] PATH_INFO: `/v3/users` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:16.074004 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f99e849-c002-4230-b470-d15d67b4696b None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81438) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:54:16.074310 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f99e849-c002-4230-b470-d15d67b4696b None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:16.075940 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f99e849-c002-4230-b470-d15d67b4696b None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:16.106613 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 316/632] 10.222.0.50 () {56 vars in 1133 bytes} [Tue Sep 13 15:54:15 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 234 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:54:16.113404 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67b794f2-e6ab-476d-825c-bafdf2e4f36b None None] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:16.189712 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f99e849-c002-4230-b470-d15d67b4696b None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:16.222674 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:54:16.269962 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67b794f2-e6ab-476d-825c-bafdf2e4f36b None None] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:16.323017 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d9c510c-c137-4717-883a-f7c328b4be8a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:16.323641 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d9c510c-c137-4717-883a-f7c328b4be8a None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:16.323918 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d9c510c-c137-4717-883a-f7c328b4be8a None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:16.324192 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d9c510c-c137-4717-883a-f7c328b4be8a None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:16.347844 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:54:16.377318 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f99e849-c002-4230-b470-d15d67b4696b None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:16.399103 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d9c510c-c137-4717-883a-f7c328b4be8a None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:16.401102 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d9c510c-c137-4717-883a-f7c328b4be8a None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:16.412110 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6099c614-d433-4cd4-8a12-3dc8d33bad7a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:16.412824 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6099c614-d433-4cd4-8a12-3dc8d33bad7a None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:16.413112 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6099c614-d433-4cd4-8a12-3dc8d33bad7a None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:16.413385 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6099c614-d433-4cd4-8a12-3dc8d33bad7a None admin] PATH_INFO: `/v3/roles/service` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:16.425962 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6099c614-d433-4cd4-8a12-3dc8d33bad7a None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 13 15:54:16.432149 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application [None req-6099c614-d433-4cd4-8a12-3dc8d33bad7a None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 13 15:54:16.432149 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:54:16.432149 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:54:16.432149 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:54:16.432149 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:54:16.432149 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:54:16.432149 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:54:16.432149 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:54:16.432149 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:54:16.432149 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:54:16.432149 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:54:16.432149 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:54:16.432149 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 13 15:54:16.432149 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 13 15:54:16.432149 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 13 15:54:16.432149 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 13 15:54:16.433357 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 13 15:54:16.433357 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 13 15:54:16.433357 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 13 15:54:16.433357 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 13 15:54:16.433357 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:54:16.433357 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:54:16.433357 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:54:16.433357 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:54:16.433357 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:54:16.433357 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:54:16.433357 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:54:16.433357 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application with Lock( Sep 13 15:54:16.433357 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:54:16.433357 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:54:16.433357 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:54:16.433357 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:54:16.434206 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:54:16.434206 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:54:16.434206 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:54:16.434206 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:54:16.434206 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 13 15:54:16.434206 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 13 15:54:16.434206 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 13 15:54:16.434206 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 13 15:54:16.434206 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 13 15:54:16.434206 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 13 15:54:16.434206 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 13 15:54:16.434206 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Sep 13 15:54:16.434206 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 317/633] 10.222.0.50 () {56 vars in 1102 bytes} [Tue Sep 13 15:54:16 2022] GET /identity/v3/roles/service => generated 85 bytes in 244 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:54:16.434955 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c65d30e-106c-4a2e-afb5-3cf50fc0b8f5 None None] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:16.434955 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c65d30e-106c-4a2e-afb5-3cf50fc0b8f5 None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:16.435696 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c65d30e-106c-4a2e-afb5-3cf50fc0b8f5 None None] PATH_INFO: `/` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:16.440819 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 318/634] 10.222.0.50 () {54 vars in 927 bytes} [Tue Sep 13 15:54:16 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:54:16.440819 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c65d30e-106c-4a2e-afb5-3cf50fc0b8f5 None None] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:16.517044 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 317/635] 10.222.0.50 () {58 vars in 1282 bytes} [Tue Sep 13 15:54:16 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 404 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 13 15:54:16.517915 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d9c510c-c137-4717-883a-f7c328b4be8a None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:16.524810 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:54:16.551187 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c65d30e-106c-4a2e-afb5-3cf50fc0b8f5 None None] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:16.579999 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:54:16.593159 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27fd9a8d-e21c-45d1-9c3d-06f3cd98b1a6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:16.593790 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27fd9a8d-e21c-45d1-9c3d-06f3cd98b1a6 None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:16.594062 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27fd9a8d-e21c-45d1-9c3d-06f3cd98b1a6 None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:16.594328 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27fd9a8d-e21c-45d1-9c3d-06f3cd98b1a6 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:16.605620 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27fd9a8d-e21c-45d1-9c3d-06f3cd98b1a6 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 13 15:54:16.610558 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application [None req-27fd9a8d-e21c-45d1-9c3d-06f3cd98b1a6 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 13 15:54:16.610558 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:54:16.610558 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:54:16.610558 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:54:16.610558 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:54:16.610558 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:54:16.610558 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:54:16.610558 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:54:16.610558 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:54:16.610558 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:54:16.610558 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:54:16.610558 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:54:16.610558 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 13 15:54:16.610558 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 13 15:54:16.610558 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 13 15:54:16.610558 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 13 15:54:16.611541 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 13 15:54:16.611541 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 13 15:54:16.611541 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 13 15:54:16.611541 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 13 15:54:16.611541 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:54:16.611541 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:54:16.611541 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:54:16.611541 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:54:16.611541 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:54:16.611541 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:54:16.611541 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:54:16.611541 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application with Lock( Sep 13 15:54:16.611541 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:54:16.611541 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:54:16.611541 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:54:16.611541 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:54:16.612426 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:54:16.612426 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:54:16.612426 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:54:16.612426 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:54:16.612426 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 13 15:54:16.612426 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 13 15:54:16.612426 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 13 15:54:16.612426 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 13 15:54:16.612426 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 13 15:54:16.612426 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 13 15:54:16.612426 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 13 15:54:16.612426 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Sep 13 15:54:16.612426 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 319/636] 10.222.0.50 () {56 vars in 1102 bytes} [Tue Sep 13 15:54:16 2022] GET /identity/v3/roles/service => generated 85 bytes in 175 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:54:16.613146 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05f3a820-2c62-4f0c-bb56-756b17b942ad None None] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:16.613146 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05f3a820-2c62-4f0c-bb56-756b17b942ad None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:16.613268 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05f3a820-2c62-4f0c-bb56-756b17b942ad None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:16.632119 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.auth.core [None req-05f3a820-2c62-4f0c-bb56-756b17b942ad None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81438) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:54:16.633320 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-05f3a820-2c62-4f0c-bb56-756b17b942ad None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:54:16.635702 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d9c510c-c137-4717-883a-f7c328b4be8a None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:16.670540 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ecf6eaa-58e4-4430-82d6-2243fa58fbf8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:16.671170 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ecf6eaa-58e4-4430-82d6-2243fa58fbf8 None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:16.671455 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ecf6eaa-58e4-4430-82d6-2243fa58fbf8 None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:16.671727 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ecf6eaa-58e4-4430-82d6-2243fa58fbf8 None admin] PATH_INFO: `/v3/roles` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:16.672387 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ecf6eaa-58e4-4430-82d6-2243fa58fbf8 None admin] RBAC: Adding query filter params (name=service) {{(pid=81437) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:54:16.672690 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ecf6eaa-58e4-4430-82d6-2243fa58fbf8 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:16.674057 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ecf6eaa-58e4-4430-82d6-2243fa58fbf8 None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:16.683502 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 318/637] 10.222.0.50 () {56 vars in 1103 bytes} [Tue Sep 13 15:54:16 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:54:16.684315 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ecf6eaa-58e4-4430-82d6-2243fa58fbf8 None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:16.712163 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 320/638] 10.222.0.50 () {58 vars in 1024 bytes} [Tue Sep 13 15:54:16 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 13 15:54:16.714255 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05f3a820-2c62-4f0c-bb56-756b17b942ad None None] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:16.806818 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ecf6eaa-58e4-4430-82d6-2243fa58fbf8 None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:16.813507 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:54:16.822463 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:54:16.844446 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9678b733-9bc1-4de5-9ed2-ab29b6fd22e0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:16.845078 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9678b733-9bc1-4de5-9ed2-ab29b6fd22e0 None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:16.845363 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9678b733-9bc1-4de5-9ed2-ab29b6fd22e0 None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:16.845636 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9678b733-9bc1-4de5-9ed2-ab29b6fd22e0 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:16.857382 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9678b733-9bc1-4de5-9ed2-ab29b6fd22e0 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:16.858845 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9678b733-9bc1-4de5-9ed2-ab29b6fd22e0 None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:16.863721 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05f3a820-2c62-4f0c-bb56-756b17b942ad None None] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:16.873437 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application [None req-9678b733-9bc1-4de5-9ed2-ab29b6fd22e0 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:54:16.873437 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:54:16.873437 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:54:16.873437 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:54:16.873437 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:54:16.873437 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:54:16.873437 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:54:16.873437 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:54:16.873437 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:54:16.873437 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:54:16.873437 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:54:16.873437 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:54:16.873437 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 13 15:54:16.873437 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 13 15:54:16.873437 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 13 15:54:16.873437 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 13 15:54:16.874415 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:54:16.874415 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:54:16.874415 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:54:16.874415 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:54:16.874415 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:54:16.874415 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:54:16.874415 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:54:16.874415 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application with Lock( Sep 13 15:54:16.874415 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:54:16.874415 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:54:16.874415 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:54:16.874415 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:54:16.874415 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:54:16.874415 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:54:16.874415 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:54:16.874415 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:54:16.874415 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 13 15:54:16.875358 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 13 15:54:16.875358 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:54:16.875358 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application Sep 13 15:54:16.875358 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 319/639] 10.222.0.50 () {56 vars in 1108 bytes} [Tue Sep 13 15:54:16 2022] GET /identity/v3/domains/Default => generated 87 bytes in 191 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:54:16.875846 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9678b733-9bc1-4de5-9ed2-ab29b6fd22e0 None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:16.911623 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-297b66a2-4d08-4fea-a60a-ff16c04b2072 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:16.912765 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-297b66a2-4d08-4fea-a60a-ff16c04b2072 None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:16.914111 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-297b66a2-4d08-4fea-a60a-ff16c04b2072 None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:16.914582 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-297b66a2-4d08-4fea-a60a-ff16c04b2072 None admin] PATH_INFO: `/v3/roles` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:16.916081 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-297b66a2-4d08-4fea-a60a-ff16c04b2072 None admin] RBAC: Adding query filter params (name=service) {{(pid=81438) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:54:16.916577 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-297b66a2-4d08-4fea-a60a-ff16c04b2072 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:16.918134 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-297b66a2-4d08-4fea-a60a-ff16c04b2072 None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:16.935869 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 321/640] 10.222.0.50 () {56 vars in 1103 bytes} [Tue Sep 13 15:54:16 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 217 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:54:16.936274 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-297b66a2-4d08-4fea-a60a-ff16c04b2072 None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:16.987382 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9678b733-9bc1-4de5-9ed2-ab29b6fd22e0 None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:17.022795 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c76b226-7ce1-4c6c-b4ed-57d501e0e5a9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:17.023462 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c76b226-7ce1-4c6c-b4ed-57d501e0e5a9 None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:17.023722 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c76b226-7ce1-4c6c-b4ed-57d501e0e5a9 None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:17.023997 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c76b226-7ce1-4c6c-b4ed-57d501e0e5a9 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:17.035032 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c76b226-7ce1-4c6c-b4ed-57d501e0e5a9 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:17.036477 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c76b226-7ce1-4c6c-b4ed-57d501e0e5a9 None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:17.051328 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application [None req-7c76b226-7ce1-4c6c-b4ed-57d501e0e5a9 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:54:17.051328 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:54:17.051328 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:54:17.051328 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:54:17.051328 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:54:17.051328 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:54:17.051328 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:54:17.051328 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:54:17.051328 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:54:17.051328 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:54:17.051328 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:54:17.051328 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:54:17.051328 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 13 15:54:17.051328 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 13 15:54:17.051328 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 13 15:54:17.051328 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 13 15:54:17.052326 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:54:17.052326 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:54:17.052326 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:54:17.052326 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:54:17.052326 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:54:17.052326 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:54:17.052326 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:54:17.052326 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application with Lock( Sep 13 15:54:17.052326 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:54:17.052326 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:54:17.052326 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:54:17.052326 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:54:17.052326 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:54:17.052326 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:54:17.052326 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:54:17.052326 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:54:17.052326 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 13 15:54:17.053256 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 13 15:54:17.053256 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:54:17.053256 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application Sep 13 15:54:17.053256 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 320/641] 10.222.0.50 () {56 vars in 1108 bytes} [Tue Sep 13 15:54:16 2022] GET /identity/v3/domains/Default => generated 87 bytes in 177 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:54:17.053500 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c76b226-7ce1-4c6c-b4ed-57d501e0e5a9 None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:17.072231 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-297b66a2-4d08-4fea-a60a-ff16c04b2072 None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:17.079702 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:54:17.101740 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:54:17.110781 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2fef438-8e4e-4dc5-9684-98200d11515a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:17.111428 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2fef438-8e4e-4dc5-9684-98200d11515a None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:17.111710 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2fef438-8e4e-4dc5-9684-98200d11515a None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:17.111984 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2fef438-8e4e-4dc5-9684-98200d11515a None admin] PATH_INFO: `/v3/domains` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:17.112714 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2fef438-8e4e-4dc5-9684-98200d11515a None admin] RBAC: Adding query filter params (name=Default) {{(pid=81438) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:54:17.113021 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2fef438-8e4e-4dc5-9684-98200d11515a None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:17.114387 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2fef438-8e4e-4dc5-9684-98200d11515a None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:17.126713 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 322/642] 10.222.0.50 () {56 vars in 1109 bytes} [Tue Sep 13 15:54:16 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 191 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:54:17.127553 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2fef438-8e4e-4dc5-9684-98200d11515a None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:17.168958 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c76b226-7ce1-4c6c-b4ed-57d501e0e5a9 None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:17.203431 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a588998-d278-4acd-9e4b-0e3436b1feda None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:17.204059 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a588998-d278-4acd-9e4b-0e3436b1feda None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:17.204339 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a588998-d278-4acd-9e4b-0e3436b1feda None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:17.204611 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a588998-d278-4acd-9e4b-0e3436b1feda None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:17.216134 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a588998-d278-4acd-9e4b-0e3436b1feda None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:17.217582 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a588998-d278-4acd-9e4b-0e3436b1feda None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:17.231627 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application [None req-4a588998-d278-4acd-9e4b-0e3436b1feda None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:54:17.231627 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:54:17.231627 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:54:17.231627 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:54:17.231627 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:54:17.231627 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:54:17.231627 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:54:17.231627 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:54:17.231627 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:54:17.231627 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:54:17.231627 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:54:17.231627 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:54:17.231627 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 13 15:54:17.231627 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 13 15:54:17.231627 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 13 15:54:17.231627 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 13 15:54:17.232864 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:54:17.232864 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:54:17.232864 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:54:17.232864 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:54:17.232864 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:54:17.232864 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:54:17.232864 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:54:17.232864 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application with Lock( Sep 13 15:54:17.232864 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:54:17.232864 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:54:17.232864 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:54:17.232864 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:54:17.232864 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:54:17.232864 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:54:17.232864 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:54:17.232864 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:54:17.232864 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 13 15:54:17.233849 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 13 15:54:17.233849 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:54:17.233849 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application Sep 13 15:54:17.237356 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 321/643] 10.222.0.50 () {56 vars in 1108 bytes} [Tue Sep 13 15:54:17 2022] GET /identity/v3/domains/Default => generated 87 bytes in 184 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:54:17.238237 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a588998-d278-4acd-9e4b-0e3436b1feda None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:17.246548 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2fef438-8e4e-4dc5-9684-98200d11515a None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:17.288597 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e74a9bb-18f4-470d-b810-0d1ec9069eb4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:17.289233 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e74a9bb-18f4-470d-b810-0d1ec9069eb4 None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:17.289519 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e74a9bb-18f4-470d-b810-0d1ec9069eb4 None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:17.289833 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e74a9bb-18f4-470d-b810-0d1ec9069eb4 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:17.300997 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e74a9bb-18f4-470d-b810-0d1ec9069eb4 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 13 15:54:17.307028 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application [None req-1e74a9bb-18f4-470d-b810-0d1ec9069eb4 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 13 15:54:17.307028 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:54:17.307028 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:54:17.307028 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:54:17.307028 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:54:17.307028 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:54:17.307028 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:54:17.307028 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:54:17.307028 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:54:17.307028 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:54:17.307028 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:54:17.307028 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:54:17.307028 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 13 15:54:17.307028 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 13 15:54:17.307028 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 13 15:54:17.307028 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 13 15:54:17.308224 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 13 15:54:17.308224 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 13 15:54:17.308224 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 13 15:54:17.308224 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 13 15:54:17.308224 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:54:17.308224 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:54:17.308224 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:54:17.308224 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:54:17.308224 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:54:17.308224 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:54:17.308224 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:54:17.308224 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application with Lock( Sep 13 15:54:17.308224 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:54:17.308224 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:54:17.308224 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:54:17.308224 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:54:17.309142 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:54:17.309142 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:54:17.309142 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:54:17.309142 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:54:17.309142 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 13 15:54:17.309142 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 13 15:54:17.309142 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 13 15:54:17.309142 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 13 15:54:17.309142 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 13 15:54:17.309142 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 13 15:54:17.309142 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 13 15:54:17.309142 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Sep 13 15:54:17.309142 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 323/644] 10.222.0.50 () {56 vars in 1102 bytes} [Tue Sep 13 15:54:17 2022] GET /identity/v3/roles/service => generated 85 bytes in 181 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:54:17.309954 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e74a9bb-18f4-470d-b810-0d1ec9069eb4 None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:17.313625 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:54:17.353002 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a588998-d278-4acd-9e4b-0e3436b1feda None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:17.370668 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:54:17.389279 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e91aea2f-0054-474d-9eb3-6da525a1931a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:17.389890 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e91aea2f-0054-474d-9eb3-6da525a1931a None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:17.393827 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e91aea2f-0054-474d-9eb3-6da525a1931a None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:17.393827 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e91aea2f-0054-474d-9eb3-6da525a1931a None admin] PATH_INFO: `/v3/domains` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:17.393827 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e91aea2f-0054-474d-9eb3-6da525a1931a None admin] RBAC: Adding query filter params (name=Default) {{(pid=81437) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:54:17.393827 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e91aea2f-0054-474d-9eb3-6da525a1931a None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:17.393827 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e91aea2f-0054-474d-9eb3-6da525a1931a None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:17.414500 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 322/645] 10.222.0.50 () {56 vars in 1109 bytes} [Tue Sep 13 15:54:17 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:54:17.415456 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e91aea2f-0054-474d-9eb3-6da525a1931a None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:17.436788 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e74a9bb-18f4-470d-b810-0d1ec9069eb4 None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:17.474115 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-598cbd89-9639-4975-9074-5826eb85131b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:17.474743 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-598cbd89-9639-4975-9074-5826eb85131b None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:17.475171 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-598cbd89-9639-4975-9074-5826eb85131b None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:17.475449 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-598cbd89-9639-4975-9074-5826eb85131b None admin] PATH_INFO: `/v3/projects/service` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:17.483179 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-598cbd89-9639-4975-9074-5826eb85131b None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:17.484636 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-598cbd89-9639-4975-9074-5826eb85131b None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:17.496792 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application [None req-598cbd89-9639-4975-9074-5826eb85131b None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 13 15:54:17.496792 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:54:17.496792 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:54:17.496792 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:54:17.496792 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:54:17.496792 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:54:17.496792 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:54:17.496792 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:54:17.496792 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:54:17.496792 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:54:17.496792 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:54:17.496792 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:54:17.496792 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 13 15:54:17.496792 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 13 15:54:17.496792 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 13 15:54:17.497856 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 13 15:54:17.497856 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:54:17.497856 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:54:17.497856 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:54:17.497856 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:54:17.497856 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:54:17.497856 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:54:17.497856 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:54:17.497856 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application with Lock( Sep 13 15:54:17.497856 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:54:17.497856 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:54:17.497856 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:54:17.497856 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:54:17.497856 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:54:17.497856 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:54:17.497856 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:54:17.497856 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:54:17.498783 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 13 15:54:17.498783 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 13 15:54:17.498783 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 13 15:54:17.498783 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 13 15:54:17.498783 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 13 15:54:17.498783 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 13 15:54:17.498783 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 13 15:54:17.498783 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Sep 13 15:54:17.498783 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 324/646] 10.222.0.50 () {56 vars in 1111 bytes} [Tue Sep 13 15:54:17 2022] GET /identity/v3/projects/service => generated 88 bytes in 189 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:54:17.499330 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-598cbd89-9639-4975-9074-5826eb85131b None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:17.524508 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e91aea2f-0054-474d-9eb3-6da525a1931a None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:17.553621 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:54:17.561065 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8351f1c2-931e-49d8-8edd-e4c524b91d8b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:17.561666 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8351f1c2-931e-49d8-8edd-e4c524b91d8b None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:17.561945 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8351f1c2-931e-49d8-8edd-e4c524b91d8b None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:17.562269 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8351f1c2-931e-49d8-8edd-e4c524b91d8b None admin] PATH_INFO: `/v3/domains` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:17.562992 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8351f1c2-931e-49d8-8edd-e4c524b91d8b None admin] RBAC: Adding query filter params (name=Default) {{(pid=81437) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:54:17.563296 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8351f1c2-931e-49d8-8edd-e4c524b91d8b None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:17.564657 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8351f1c2-931e-49d8-8edd-e4c524b91d8b None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:17.580529 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 323/647] 10.222.0.50 () {56 vars in 1109 bytes} [Tue Sep 13 15:54:17 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:54:17.580529 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8351f1c2-931e-49d8-8edd-e4c524b91d8b None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:17.623789 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-598cbd89-9639-4975-9074-5826eb85131b None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:17.633025 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:54:17.661640 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7fd95e0-666d-49c5-9bff-ffb2f5366efa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:17.662313 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7fd95e0-666d-49c5-9bff-ffb2f5366efa None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:17.662600 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7fd95e0-666d-49c5-9bff-ffb2f5366efa None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:17.662877 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7fd95e0-666d-49c5-9bff-ffb2f5366efa None admin] PATH_INFO: `/v3/roles` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:17.663562 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7fd95e0-666d-49c5-9bff-ffb2f5366efa None admin] RBAC: Adding query filter params (name=service) {{(pid=81438) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:54:17.663874 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7fd95e0-666d-49c5-9bff-ffb2f5366efa None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:17.665241 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7fd95e0-666d-49c5-9bff-ffb2f5366efa None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:17.674019 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 325/648] 10.222.0.50 () {56 vars in 1103 bytes} [Tue Sep 13 15:54:17 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:54:17.674831 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7fd95e0-666d-49c5-9bff-ffb2f5366efa None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:17.691974 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8351f1c2-931e-49d8-8edd-e4c524b91d8b None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:17.731557 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd57db2a-daea-4376-b632-9d37091dcfe7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:17.732250 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd57db2a-daea-4376-b632-9d37091dcfe7 None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:17.732535 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd57db2a-daea-4376-b632-9d37091dcfe7 None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:17.732811 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd57db2a-daea-4376-b632-9d37091dcfe7 None admin] PATH_INFO: `/v3/users/glance` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:17.740625 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd57db2a-daea-4376-b632-9d37091dcfe7 None admin] RBAC: Authorizing `identity:get_user(user_id=glance)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:17.742076 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd57db2a-daea-4376-b632-9d37091dcfe7 None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:17.755286 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application [None req-dd57db2a-daea-4376-b632-9d37091dcfe7 None admin] Could not find user: glance.: keystone.exception.UserNotFound: Could not find user: glance. Sep 13 15:54:17.755286 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:54:17.755286 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:54:17.755286 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:54:17.755286 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:54:17.755286 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:54:17.755286 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:54:17.755286 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:54:17.755286 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:54:17.755286 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:54:17.755286 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:54:17.755286 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:54:17.755286 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 13 15:54:17.755286 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 13 15:54:17.755286 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 13 15:54:17.755286 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 13 15:54:17.756239 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:54:17.756239 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:54:17.756239 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 13 15:54:17.756239 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 13 15:54:17.756239 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 13 15:54:17.756239 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 13 15:54:17.756239 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:54:17.756239 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:54:17.756239 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:54:17.756239 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:54:17.756239 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:54:17.756239 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application with Lock( Sep 13 15:54:17.756239 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:54:17.756239 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:54:17.756239 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:54:17.756239 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:54:17.756239 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:54:17.756239 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:54:17.757183 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:54:17.757183 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:54:17.757183 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 13 15:54:17.757183 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 13 15:54:17.757183 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 13 15:54:17.757183 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 13 15:54:17.757183 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 13 15:54:17.757183 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 13 15:54:17.757183 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance. Sep 13 15:54:17.757183 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application Sep 13 15:54:17.757183 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 324/649] 10.222.0.50 () {56 vars in 1099 bytes} [Tue Sep 13 15:54:17 2022] GET /identity/v3/users/glance => generated 84 bytes in 180 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:54:17.757791 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd57db2a-daea-4376-b632-9d37091dcfe7 None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:17.789391 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7fd95e0-666d-49c5-9bff-ffb2f5366efa None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:17.812648 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:54:17.832021 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3a58936-f01e-49c0-a613-fb12e40d831e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:17.832985 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3a58936-f01e-49c0-a613-fb12e40d831e None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:17.833265 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3a58936-f01e-49c0-a613-fb12e40d831e None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:17.833538 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3a58936-f01e-49c0-a613-fb12e40d831e None admin] PATH_INFO: `/v3/projects` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:17.834235 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3a58936-f01e-49c0-a613-fb12e40d831e None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81438) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:54:17.834541 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3a58936-f01e-49c0-a613-fb12e40d831e None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:17.836001 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3a58936-f01e-49c0-a613-fb12e40d831e None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:17.848557 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 326/650] 10.222.0.50 () {56 vars in 1148 bytes} [Tue Sep 13 15:54:17 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:54:17.849383 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3a58936-f01e-49c0-a613-fb12e40d831e None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:17.873884 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd57db2a-daea-4376-b632-9d37091dcfe7 None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:17.884638 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:54:17.918439 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-777358ab-cc8f-4f91-b7df-8576ac70e56a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:17.919196 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-777358ab-cc8f-4f91-b7df-8576ac70e56a None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:17.919482 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-777358ab-cc8f-4f91-b7df-8576ac70e56a None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:17.919761 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-777358ab-cc8f-4f91-b7df-8576ac70e56a None admin] PATH_INFO: `/v3/users/cinder` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:17.927429 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-777358ab-cc8f-4f91-b7df-8576ac70e56a None admin] RBAC: Authorizing `identity:get_user(user_id=cinder)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:17.928935 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-777358ab-cc8f-4f91-b7df-8576ac70e56a None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:17.941022 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application [None req-777358ab-cc8f-4f91-b7df-8576ac70e56a None admin] Could not find user: cinder.: keystone.exception.UserNotFound: Could not find user: cinder. Sep 13 15:54:17.941022 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:54:17.941022 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:54:17.941022 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:54:17.941022 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:54:17.941022 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:54:17.941022 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:54:17.941022 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:54:17.941022 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:54:17.941022 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:54:17.941022 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:54:17.941022 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:54:17.941022 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 13 15:54:17.941022 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 13 15:54:17.941022 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 13 15:54:17.941022 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 13 15:54:17.942008 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:54:17.942008 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:54:17.942008 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 13 15:54:17.942008 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 13 15:54:17.942008 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 13 15:54:17.942008 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 13 15:54:17.942008 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:54:17.942008 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:54:17.942008 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:54:17.942008 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:54:17.942008 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:54:17.942008 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application with Lock( Sep 13 15:54:17.942008 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:54:17.942008 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:54:17.942008 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:54:17.942008 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:54:17.942008 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:54:17.942008 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:54:17.943005 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:54:17.943005 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:54:17.943005 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 13 15:54:17.943005 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 13 15:54:17.943005 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 13 15:54:17.943005 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 13 15:54:17.943005 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 13 15:54:17.943005 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 13 15:54:17.943005 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: cinder. Sep 13 15:54:17.943005 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application Sep 13 15:54:17.943005 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 325/651] 10.222.0.50 () {56 vars in 1099 bytes} [Tue Sep 13 15:54:17 2022] GET /identity/v3/users/cinder => generated 84 bytes in 185 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:54:17.943005 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-777358ab-cc8f-4f91-b7df-8576ac70e56a None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:17.964983 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3a58936-f01e-49c0-a613-fb12e40d831e None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:18.000469 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-638c320b-0196-444c-a8da-5ad86fe323ee None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:18.001310 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-638c320b-0196-444c-a8da-5ad86fe323ee None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:18.001675 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-638c320b-0196-444c-a8da-5ad86fe323ee None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:18.002164 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-638c320b-0196-444c-a8da-5ad86fe323ee None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:18.013331 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-638c320b-0196-444c-a8da-5ad86fe323ee None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:18.014713 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-638c320b-0196-444c-a8da-5ad86fe323ee None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:18.030346 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application [None req-638c320b-0196-444c-a8da-5ad86fe323ee None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:54:18.030346 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:54:18.030346 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:54:18.030346 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:54:18.030346 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:54:18.030346 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:54:18.030346 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:54:18.030346 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:54:18.030346 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:54:18.030346 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:54:18.030346 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:54:18.030346 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:54:18.030346 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 13 15:54:18.030346 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 13 15:54:18.030346 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 13 15:54:18.030346 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 13 15:54:18.031659 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:54:18.031659 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:54:18.031659 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:54:18.031659 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:54:18.031659 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:54:18.031659 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:54:18.031659 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:54:18.031659 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application with Lock( Sep 13 15:54:18.031659 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:54:18.031659 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:54:18.031659 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:54:18.031659 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:54:18.031659 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:54:18.031659 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:54:18.031659 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:54:18.031659 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:54:18.031659 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 13 15:54:18.033018 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 13 15:54:18.033018 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:54:18.033018 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Sep 13 15:54:18.033018 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 327/652] 10.222.0.50 () {56 vars in 1108 bytes} [Tue Sep 13 15:54:17 2022] GET /identity/v3/domains/Default => generated 87 bytes in 183 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:54:18.033018 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-638c320b-0196-444c-a8da-5ad86fe323ee None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:18.067209 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-777358ab-cc8f-4f91-b7df-8576ac70e56a None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:18.078531 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:54:18.106864 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f03248fe-94fb-419b-8c43-4db88b46e62b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:18.107800 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f03248fe-94fb-419b-8c43-4db88b46e62b None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:18.108083 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f03248fe-94fb-419b-8c43-4db88b46e62b None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:18.108363 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f03248fe-94fb-419b-8c43-4db88b46e62b None admin] PATH_INFO: `/v3/users` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:18.109293 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f03248fe-94fb-419b-8c43-4db88b46e62b None admin] RBAC: Adding query filter params (domain_id=default, name=glance) {{(pid=81437) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:54:18.109975 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f03248fe-94fb-419b-8c43-4db88b46e62b None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:18.111788 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f03248fe-94fb-419b-8c43-4db88b46e62b None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:18.135031 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 326/653] 10.222.0.50 () {56 vars in 1137 bytes} [Tue Sep 13 15:54:17 2022] GET /identity/v3/users?domain_id=default&name=glance => generated 372 bytes in 192 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:54:18.136499 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f03248fe-94fb-419b-8c43-4db88b46e62b None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:18.144519 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:54:18.167888 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-638c320b-0196-444c-a8da-5ad86fe323ee None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:18.204545 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1da6f09e-48d6-40c0-acc7-7ed8beb72a35 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:18.205502 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1da6f09e-48d6-40c0-acc7-7ed8beb72a35 None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:18.206037 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1da6f09e-48d6-40c0-acc7-7ed8beb72a35 None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:18.206505 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1da6f09e-48d6-40c0-acc7-7ed8beb72a35 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:18.207436 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1da6f09e-48d6-40c0-acc7-7ed8beb72a35 None admin] RBAC: Adding query filter params (role.id=da847124d9374ac88b27968451c486ab, scope.project.id=5811e3b83a1e462e9320cad79a3ad791, user.id=4e039b46c5e3445bbb39904b00c5430e) {{(pid=81438) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:54:18.207926 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1da6f09e-48d6-40c0-acc7-7ed8beb72a35 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:18.209526 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1da6f09e-48d6-40c0-acc7-7ed8beb72a35 None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:18.216452 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 328/654] 10.222.0.50 () {56 vars in 1374 bytes} [Tue Sep 13 15:54:18 2022] GET /identity/v3/role_assignments?user.id=4e039b46c5e3445bbb39904b00c5430e&scope.project.id=5811e3b83a1e462e9320cad79a3ad791&role.id=da847124d9374ac88b27968451c486ab => generated 608 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:54:18.217266 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1da6f09e-48d6-40c0-acc7-7ed8beb72a35 None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:18.261466 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f03248fe-94fb-419b-8c43-4db88b46e62b None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:18.303663 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8918f89-edd8-4e34-bfb7-9cb9c2c9635c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:18.304601 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8918f89-edd8-4e34-bfb7-9cb9c2c9635c None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:18.304880 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8918f89-edd8-4e34-bfb7-9cb9c2c9635c None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:18.305204 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8918f89-edd8-4e34-bfb7-9cb9c2c9635c None admin] PATH_INFO: `/v3/users` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:18.306213 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8918f89-edd8-4e34-bfb7-9cb9c2c9635c None admin] RBAC: Adding query filter params (domain_id=default, name=cinder) {{(pid=81437) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:54:18.306359 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8918f89-edd8-4e34-bfb7-9cb9c2c9635c None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:18.308033 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8918f89-edd8-4e34-bfb7-9cb9c2c9635c None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:18.334485 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1da6f09e-48d6-40c0-acc7-7ed8beb72a35 None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:18.335401 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 327/655] 10.222.0.50 () {56 vars in 1137 bytes} [Tue Sep 13 15:54:18 2022] GET /identity/v3/users?domain_id=default&name=cinder => generated 372 bytes in 199 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:54:18.336374 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8918f89-edd8-4e34-bfb7-9cb9c2c9635c None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:18.366677 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:54:18.375479 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a598ed34-7798-4418-b6da-38f24dd4af2e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:18.376202 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a598ed34-7798-4418-b6da-38f24dd4af2e None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:18.376487 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a598ed34-7798-4418-b6da-38f24dd4af2e None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:18.376759 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a598ed34-7798-4418-b6da-38f24dd4af2e None admin] PATH_INFO: `/v3/domains` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:18.377411 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a598ed34-7798-4418-b6da-38f24dd4af2e None admin] RBAC: Adding query filter params (name=Default) {{(pid=81438) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:54:18.377715 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a598ed34-7798-4418-b6da-38f24dd4af2e None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:18.379124 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a598ed34-7798-4418-b6da-38f24dd4af2e None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:18.396814 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 329/656] 10.222.0.50 () {56 vars in 1109 bytes} [Tue Sep 13 15:54:18 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:54:18.396814 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d91846e2-a589-422c-b189-86ff81ee5247 None None] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:18.396814 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d91846e2-a589-422c-b189-86ff81ee5247 None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:18.396814 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d91846e2-a589-422c-b189-86ff81ee5247 None None] PATH_INFO: `/` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:18.396814 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 330/657] 10.222.0.50 () {54 vars in 927 bytes} [Tue Sep 13 15:54:18 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:54:18.396814 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d91846e2-a589-422c-b189-86ff81ee5247 None None] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:18.406554 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:54:18.465181 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8918f89-edd8-4e34-bfb7-9cb9c2c9635c None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:18.501367 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a5fb298-853c-41b6-9c7a-a4b4f0b730cf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:18.501978 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a5fb298-853c-41b6-9c7a-a4b4f0b730cf None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:18.502259 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a5fb298-853c-41b6-9c7a-a4b4f0b730cf None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:18.502531 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a5fb298-853c-41b6-9c7a-a4b4f0b730cf None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:18.525182 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d91846e2-a589-422c-b189-86ff81ee5247 None None] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:18.560775 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-168960cd-49e4-41d5-987a-b9e81dcdeed4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:18.561306 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-168960cd-49e4-41d5-987a-b9e81dcdeed4 None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:18.561592 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-168960cd-49e4-41d5-987a-b9e81dcdeed4 None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:18.561871 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-168960cd-49e4-41d5-987a-b9e81dcdeed4 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:18.576022 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a5fb298-853c-41b6-9c7a-a4b4f0b730cf None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:18.577675 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a5fb298-853c-41b6-9c7a-a4b4f0b730cf None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:18.606968 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:54:18.638599 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-168960cd-49e4-41d5-987a-b9e81dcdeed4 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:18.640442 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-168960cd-49e4-41d5-987a-b9e81dcdeed4 None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:18.648902 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:54:18.685992 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 328/658] 10.222.0.50 () {58 vars in 1282 bytes} [Tue Sep 13 15:54:18 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 350 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 13 15:54:18.686824 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a5fb298-853c-41b6-9c7a-a4b4f0b730cf None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:18.758491 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 331/659] 10.222.0.50 () {58 vars in 1282 bytes} [Tue Sep 13 15:54:18 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 362 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 13 15:54:18.760203 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-355e7683-669e-4c3e-8669-d9ecaf353a8b None None] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:18.760708 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-355e7683-669e-4c3e-8669-d9ecaf353a8b None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:18.761190 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-355e7683-669e-4c3e-8669-d9ecaf353a8b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:18.779291 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.auth.core [None req-355e7683-669e-4c3e-8669-d9ecaf353a8b None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81438) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:54:18.780488 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-355e7683-669e-4c3e-8669-d9ecaf353a8b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:54:18.796684 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a5fb298-853c-41b6-9c7a-a4b4f0b730cf None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:18.824872 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:54:18.835436 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-194e37af-ddbe-481d-99f4-1bce0aa8ffc7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:18.836066 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-194e37af-ddbe-481d-99f4-1bce0aa8ffc7 None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:18.836342 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-194e37af-ddbe-481d-99f4-1bce0aa8ffc7 None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:18.836613 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-194e37af-ddbe-481d-99f4-1bce0aa8ffc7 None admin] PATH_INFO: `/v3/users/neutron` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:18.844254 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-194e37af-ddbe-481d-99f4-1bce0aa8ffc7 None admin] RBAC: Authorizing `identity:get_user(user_id=neutron)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:18.845824 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-194e37af-ddbe-481d-99f4-1bce0aa8ffc7 None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:18.858933 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 332/660] 10.222.0.50 () {58 vars in 1024 bytes} [Tue Sep 13 15:54:18 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 13 15:54:18.858933 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-355e7683-669e-4c3e-8669-d9ecaf353a8b None None] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:18.868200 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application [None req-194e37af-ddbe-481d-99f4-1bce0aa8ffc7 None admin] Could not find user: neutron.: keystone.exception.UserNotFound: Could not find user: neutron. Sep 13 15:54:18.868200 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:54:18.868200 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:54:18.868200 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:54:18.868200 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:54:18.868200 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:54:18.868200 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:54:18.868200 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:54:18.868200 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:54:18.868200 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:54:18.868200 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:54:18.868200 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:54:18.868200 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 13 15:54:18.868200 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 13 15:54:18.868200 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 13 15:54:18.868200 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 13 15:54:18.869244 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:54:18.869244 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:54:18.869244 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 13 15:54:18.869244 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 13 15:54:18.869244 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 13 15:54:18.869244 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 13 15:54:18.869244 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:54:18.869244 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:54:18.869244 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:54:18.869244 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:54:18.869244 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:54:18.869244 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application with Lock( Sep 13 15:54:18.869244 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:54:18.869244 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:54:18.869244 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:54:18.869244 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:54:18.869244 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:54:18.869244 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:54:18.870199 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:54:18.870199 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:54:18.870199 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 13 15:54:18.870199 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 13 15:54:18.870199 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 13 15:54:18.870199 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 13 15:54:18.870199 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 13 15:54:18.870199 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 13 15:54:18.870199 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: neutron. Sep 13 15:54:18.870199 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application Sep 13 15:54:18.874189 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 329/661] 10.222.0.50 () {56 vars in 1102 bytes} [Tue Sep 13 15:54:18 2022] GET /identity/v3/users/neutron => generated 85 bytes in 187 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:54:18.875085 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-194e37af-ddbe-481d-99f4-1bce0aa8ffc7 None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:18.904467 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:54:18.979720 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-355e7683-669e-4c3e-8669-d9ecaf353a8b None None] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:18.983642 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-194e37af-ddbe-481d-99f4-1bce0aa8ffc7 None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:19.019464 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d667a9b0-52df-4593-8c65-167b4731ea55 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:19.020399 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8ebedc2-176c-4c25-804a-8fec562a63a8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:19.020906 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d667a9b0-52df-4593-8c65-167b4731ea55 None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:19.021151 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8ebedc2-176c-4c25-804a-8fec562a63a8 None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:19.021438 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8ebedc2-176c-4c25-804a-8fec562a63a8 None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:19.021714 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8ebedc2-176c-4c25-804a-8fec562a63a8 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:19.022030 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d667a9b0-52df-4593-8c65-167b4731ea55 None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:19.022500 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d667a9b0-52df-4593-8c65-167b4731ea55 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:19.033052 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8ebedc2-176c-4c25-804a-8fec562a63a8 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:19.034255 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d667a9b0-52df-4593-8c65-167b4731ea55 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:19.034521 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8ebedc2-176c-4c25-804a-8fec562a63a8 None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:19.036432 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d667a9b0-52df-4593-8c65-167b4731ea55 None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:19.050530 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application [None req-e8ebedc2-176c-4c25-804a-8fec562a63a8 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:54:19.050530 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:54:19.050530 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:54:19.050530 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:54:19.050530 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:54:19.050530 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:54:19.050530 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:54:19.050530 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:54:19.050530 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:54:19.050530 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:54:19.050530 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:54:19.050530 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:54:19.050530 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 13 15:54:19.050530 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 13 15:54:19.050530 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 13 15:54:19.050530 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 13 15:54:19.051595 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:54:19.051595 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:54:19.051595 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:54:19.051595 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:54:19.051595 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:54:19.051595 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:54:19.051595 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:54:19.051595 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application with Lock( Sep 13 15:54:19.051595 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:54:19.051595 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:54:19.051595 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:54:19.051595 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:54:19.051595 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:54:19.051595 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:54:19.051595 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:54:19.051595 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:54:19.051595 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 13 15:54:19.052503 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 13 15:54:19.052503 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:54:19.052503 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application Sep 13 15:54:19.052503 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 330/662] 10.222.0.50 () {56 vars in 1108 bytes} [Tue Sep 13 15:54:18 2022] GET /identity/v3/domains/Default => generated 87 bytes in 177 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:54:19.053165 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8ebedc2-176c-4c25-804a-8fec562a63a8 None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:19.072553 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application [None req-d667a9b0-52df-4593-8c65-167b4731ea55 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:54:19.072553 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:54:19.072553 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:54:19.072553 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:54:19.072553 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:54:19.072553 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:54:19.072553 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:54:19.072553 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:54:19.072553 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:54:19.072553 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:54:19.072553 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:54:19.072553 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:54:19.072553 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 13 15:54:19.072553 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 13 15:54:19.072553 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 13 15:54:19.072553 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 13 15:54:19.073622 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:54:19.073622 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:54:19.073622 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:54:19.073622 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:54:19.073622 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:54:19.073622 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:54:19.073622 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:54:19.073622 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application with Lock( Sep 13 15:54:19.073622 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:54:19.073622 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:54:19.073622 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:54:19.073622 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:54:19.073622 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:54:19.073622 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:54:19.073622 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:54:19.073622 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:54:19.073622 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 13 15:54:19.074548 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 13 15:54:19.074548 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:54:19.074548 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Sep 13 15:54:19.077316 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 333/663] 10.222.0.50 () {56 vars in 1108 bytes} [Tue Sep 13 15:54:18 2022] GET /identity/v3/domains/Default => generated 87 bytes in 225 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:54:19.078496 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d667a9b0-52df-4593-8c65-167b4731ea55 None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:19.100550 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:54:19.188744 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8ebedc2-176c-4c25-804a-8fec562a63a8 None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:19.188954 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:54:19.202642 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d667a9b0-52df-4593-8c65-167b4731ea55 None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:19.225308 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a86586c-8eab-4575-9bd0-a77b77972d78 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:19.225918 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a86586c-8eab-4575-9bd0-a77b77972d78 None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:19.226197 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a86586c-8eab-4575-9bd0-a77b77972d78 None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:19.226470 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a86586c-8eab-4575-9bd0-a77b77972d78 None admin] PATH_INFO: `/v3/users` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:19.227218 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a86586c-8eab-4575-9bd0-a77b77972d78 None admin] RBAC: Adding query filter params (domain_id=default, name=neutron) {{(pid=81437) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:54:19.227522 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a86586c-8eab-4575-9bd0-a77b77972d78 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:19.229016 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a86586c-8eab-4575-9bd0-a77b77972d78 None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:19.238363 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eda2ca6f-8aa2-4e1f-ac5e-ab56b7a889c6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:19.238363 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eda2ca6f-8aa2-4e1f-ac5e-ab56b7a889c6 None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:19.238605 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eda2ca6f-8aa2-4e1f-ac5e-ab56b7a889c6 None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:19.238798 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eda2ca6f-8aa2-4e1f-ac5e-ab56b7a889c6 None admin] PATH_INFO: `/v3/domains` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:19.239484 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eda2ca6f-8aa2-4e1f-ac5e-ab56b7a889c6 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81438) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:54:19.239789 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eda2ca6f-8aa2-4e1f-ac5e-ab56b7a889c6 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:19.241140 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eda2ca6f-8aa2-4e1f-ac5e-ab56b7a889c6 None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:19.251755 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 331/664] 10.222.0.50 () {56 vars in 1139 bytes} [Tue Sep 13 15:54:19 2022] GET /identity/v3/users?domain_id=default&name=neutron => generated 374 bytes in 195 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:54:19.251755 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a86586c-8eab-4575-9bd0-a77b77972d78 None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:19.257616 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 334/665] 10.222.0.50 () {56 vars in 1109 bytes} [Tue Sep 13 15:54:19 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:54:19.260455 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eda2ca6f-8aa2-4e1f-ac5e-ab56b7a889c6 None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:19.350310 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:54:19.356443 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a86586c-8eab-4575-9bd0-a77b77972d78 None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:19.375771 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eda2ca6f-8aa2-4e1f-ac5e-ab56b7a889c6 None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:19.391284 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bbc5a7d-2196-4204-ad6a-f631e6ad6ab0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:19.391895 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bbc5a7d-2196-4204-ad6a-f631e6ad6ab0 None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:19.392175 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bbc5a7d-2196-4204-ad6a-f631e6ad6ab0 None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:19.392472 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bbc5a7d-2196-4204-ad6a-f631e6ad6ab0 None admin] PATH_INFO: `/v3/domains` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:19.393095 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bbc5a7d-2196-4204-ad6a-f631e6ad6ab0 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81437) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:54:19.393400 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bbc5a7d-2196-4204-ad6a-f631e6ad6ab0 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:19.394802 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bbc5a7d-2196-4204-ad6a-f631e6ad6ab0 None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:19.407168 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 332/666] 10.222.0.50 () {56 vars in 1109 bytes} [Tue Sep 13 15:54:19 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:54:19.407986 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bbc5a7d-2196-4204-ad6a-f631e6ad6ab0 None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:19.421811 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfc1d7bf-dd6a-43da-b5d5-b34ca57b8b7e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:19.422657 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfc1d7bf-dd6a-43da-b5d5-b34ca57b8b7e None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:19.423129 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfc1d7bf-dd6a-43da-b5d5-b34ca57b8b7e None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:19.423649 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfc1d7bf-dd6a-43da-b5d5-b34ca57b8b7e None admin] PATH_INFO: `/v3/roles/service` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:19.435086 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dfc1d7bf-dd6a-43da-b5d5-b34ca57b8b7e None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 13 15:54:19.439878 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application [None req-dfc1d7bf-dd6a-43da-b5d5-b34ca57b8b7e None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 13 15:54:19.439878 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:54:19.439878 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:54:19.439878 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:54:19.439878 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:54:19.439878 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:54:19.439878 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:54:19.439878 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:54:19.439878 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:54:19.439878 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:54:19.439878 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:54:19.439878 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:54:19.439878 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 13 15:54:19.439878 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 13 15:54:19.439878 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 13 15:54:19.439878 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 13 15:54:19.440799 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 13 15:54:19.440799 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 13 15:54:19.440799 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 13 15:54:19.440799 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 13 15:54:19.440799 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:54:19.440799 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:54:19.440799 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:54:19.440799 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:54:19.440799 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:54:19.440799 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:54:19.440799 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:54:19.440799 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application with Lock( Sep 13 15:54:19.440799 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:54:19.440799 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:54:19.440799 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:54:19.440799 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:54:19.441655 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:54:19.441655 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:54:19.441655 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:54:19.441655 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:54:19.441655 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 13 15:54:19.441655 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 13 15:54:19.441655 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 13 15:54:19.441655 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 13 15:54:19.441655 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 13 15:54:19.441655 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 13 15:54:19.441655 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 13 15:54:19.441655 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Sep 13 15:54:19.441655 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 335/667] 10.222.0.50 () {56 vars in 1102 bytes} [Tue Sep 13 15:54:19 2022] GET /identity/v3/roles/service => generated 85 bytes in 181 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:54:19.442363 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfc1d7bf-dd6a-43da-b5d5-b34ca57b8b7e None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:19.454941 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:54:19.516953 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bbc5a7d-2196-4204-ad6a-f631e6ad6ab0 None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:19.552887 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfc1d7bf-dd6a-43da-b5d5-b34ca57b8b7e None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:19.553087 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eeda14f5-bdc9-41d7-904e-763a9029610e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:19.553682 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eeda14f5-bdc9-41d7-904e-763a9029610e None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:19.553963 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eeda14f5-bdc9-41d7-904e-763a9029610e None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:19.554239 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eeda14f5-bdc9-41d7-904e-763a9029610e None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:19.581076 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:54:19.591201 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43480c4d-1ba0-4d41-a1f2-ebf3966af753 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:19.591838 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43480c4d-1ba0-4d41-a1f2-ebf3966af753 None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:19.592123 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43480c4d-1ba0-4d41-a1f2-ebf3966af753 None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:19.592399 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43480c4d-1ba0-4d41-a1f2-ebf3966af753 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:19.599575 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43480c4d-1ba0-4d41-a1f2-ebf3966af753 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:19.601067 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43480c4d-1ba0-4d41-a1f2-ebf3966af753 None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:19.612453 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application [None req-43480c4d-1ba0-4d41-a1f2-ebf3966af753 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 13 15:54:19.612453 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:54:19.612453 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:54:19.612453 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:54:19.612453 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:54:19.612453 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:54:19.612453 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:54:19.612453 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:54:19.612453 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:54:19.612453 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:54:19.612453 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:54:19.612453 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:54:19.612453 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 13 15:54:19.612453 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 13 15:54:19.612453 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 13 15:54:19.613480 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 13 15:54:19.613480 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:54:19.613480 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:54:19.613480 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:54:19.613480 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:54:19.613480 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:54:19.613480 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:54:19.613480 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:54:19.613480 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application with Lock( Sep 13 15:54:19.613480 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:54:19.613480 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:54:19.613480 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:54:19.613480 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:54:19.613480 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:54:19.613480 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:54:19.613480 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:54:19.613480 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:54:19.614379 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 13 15:54:19.614379 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 13 15:54:19.614379 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 13 15:54:19.614379 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 13 15:54:19.614379 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 13 15:54:19.614379 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 13 15:54:19.614379 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 13 15:54:19.614379 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Sep 13 15:54:19.614379 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 336/668] 10.222.0.50 () {56 vars in 1111 bytes} [Tue Sep 13 15:54:19 2022] GET /identity/v3/projects/service => generated 88 bytes in 172 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:54:19.614379 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43480c4d-1ba0-4d41-a1f2-ebf3966af753 None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:19.635601 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eeda14f5-bdc9-41d7-904e-763a9029610e None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:19.637800 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eeda14f5-bdc9-41d7-904e-763a9029610e None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:19.689680 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:54:19.729845 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43480c4d-1ba0-4d41-a1f2-ebf3966af753 None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:19.755039 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 333/669] 10.222.0.50 () {58 vars in 1282 bytes} [Tue Sep 13 15:54:19 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 347 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 13 15:54:19.756122 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eeda14f5-bdc9-41d7-904e-763a9029610e None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:19.778798 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59d49fc4-a1a3-4559-8c5a-30f5b776be58 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:19.779750 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59d49fc4-a1a3-4559-8c5a-30f5b776be58 None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:19.780031 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59d49fc4-a1a3-4559-8c5a-30f5b776be58 None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:19.780309 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59d49fc4-a1a3-4559-8c5a-30f5b776be58 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:19.789774 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59d49fc4-a1a3-4559-8c5a-30f5b776be58 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:19.792328 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59d49fc4-a1a3-4559-8c5a-30f5b776be58 None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:19.806930 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application [None req-59d49fc4-a1a3-4559-8c5a-30f5b776be58 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 13 15:54:19.806930 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:54:19.806930 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:54:19.806930 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:54:19.806930 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:54:19.806930 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:54:19.806930 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:54:19.806930 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:54:19.806930 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:54:19.806930 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:54:19.806930 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:54:19.806930 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:54:19.806930 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 13 15:54:19.806930 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 13 15:54:19.806930 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 13 15:54:19.808244 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 13 15:54:19.808244 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:54:19.808244 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:54:19.808244 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:54:19.808244 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:54:19.808244 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:54:19.808244 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:54:19.808244 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:54:19.808244 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application with Lock( Sep 13 15:54:19.808244 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:54:19.808244 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:54:19.808244 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:54:19.808244 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:54:19.808244 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:54:19.808244 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:54:19.808244 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:54:19.808244 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:54:19.809147 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 13 15:54:19.809147 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 13 15:54:19.809147 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 13 15:54:19.809147 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 13 15:54:19.809147 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 13 15:54:19.809147 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 13 15:54:19.809147 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 13 15:54:19.809147 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Sep 13 15:54:19.813578 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 337/670] 10.222.0.50 () {56 vars in 1111 bytes} [Tue Sep 13 15:54:19 2022] GET /identity/v3/projects/service => generated 88 bytes in 199 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:54:19.814454 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59d49fc4-a1a3-4559-8c5a-30f5b776be58 None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:19.836061 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:54:19.892771 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eeda14f5-bdc9-41d7-904e-763a9029610e None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:19.944621 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa067367-dea0-4eef-9f4a-a4c8b2fffdd9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:19.945237 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa067367-dea0-4eef-9f4a-a4c8b2fffdd9 None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:19.945590 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa067367-dea0-4eef-9f4a-a4c8b2fffdd9 None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:19.945796 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa067367-dea0-4eef-9f4a-a4c8b2fffdd9 None admin] PATH_INFO: `/v3/roles` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:19.946505 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa067367-dea0-4eef-9f4a-a4c8b2fffdd9 None admin] RBAC: Adding query filter params (name=service) {{(pid=81437) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:54:19.947144 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa067367-dea0-4eef-9f4a-a4c8b2fffdd9 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:19.948504 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa067367-dea0-4eef-9f4a-a4c8b2fffdd9 None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:19.954145 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59d49fc4-a1a3-4559-8c5a-30f5b776be58 None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:19.958261 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 334/671] 10.222.0.50 () {56 vars in 1103 bytes} [Tue Sep 13 15:54:19 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 202 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:54:19.959298 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa067367-dea0-4eef-9f4a-a4c8b2fffdd9 None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:19.989039 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:54:19.999445 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94e68466-c09f-4bfb-b819-04b9f3c420f9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:20.000335 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94e68466-c09f-4bfb-b819-04b9f3c420f9 None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:20.000883 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94e68466-c09f-4bfb-b819-04b9f3c420f9 None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:20.001360 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94e68466-c09f-4bfb-b819-04b9f3c420f9 None admin] PATH_INFO: `/v3/projects` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:20.002050 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94e68466-c09f-4bfb-b819-04b9f3c420f9 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81438) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:54:20.002357 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94e68466-c09f-4bfb-b819-04b9f3c420f9 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:20.004237 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94e68466-c09f-4bfb-b819-04b9f3c420f9 None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:20.021713 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 338/672] 10.222.0.50 () {56 vars in 1148 bytes} [Tue Sep 13 15:54:19 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 203 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:54:20.021713 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94e68466-c09f-4bfb-b819-04b9f3c420f9 None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:20.127907 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa067367-dea0-4eef-9f4a-a4c8b2fffdd9 None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:20.141272 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94e68466-c09f-4bfb-b819-04b9f3c420f9 None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:20.154453 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:54:20.172387 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9b53d14-473d-4227-9ee3-6fa758e62b84 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:20.173027 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9b53d14-473d-4227-9ee3-6fa758e62b84 None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:20.173304 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9b53d14-473d-4227-9ee3-6fa758e62b84 None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:20.173573 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9b53d14-473d-4227-9ee3-6fa758e62b84 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:20.182226 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-026645f1-9ba1-4a57-bc45-5a396de2c622 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:20.183602 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-026645f1-9ba1-4a57-bc45-5a396de2c622 None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:20.183929 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-026645f1-9ba1-4a57-bc45-5a396de2c622 None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:20.184226 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-026645f1-9ba1-4a57-bc45-5a396de2c622 None admin] PATH_INFO: `/v3/projects` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:20.184836 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-026645f1-9ba1-4a57-bc45-5a396de2c622 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81438) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:54:20.184939 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9b53d14-473d-4227-9ee3-6fa758e62b84 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:20.185142 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-026645f1-9ba1-4a57-bc45-5a396de2c622 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:20.186352 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9b53d14-473d-4227-9ee3-6fa758e62b84 None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:20.186615 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-026645f1-9ba1-4a57-bc45-5a396de2c622 None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:20.204412 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 339/673] 10.222.0.50 () {56 vars in 1148 bytes} [Tue Sep 13 15:54:20 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:54:20.204412 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-026645f1-9ba1-4a57-bc45-5a396de2c622 None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:20.204688 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application [None req-c9b53d14-473d-4227-9ee3-6fa758e62b84 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:54:20.204688 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:54:20.204688 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:54:20.204688 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:54:20.204688 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:54:20.204688 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:54:20.204688 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:54:20.204688 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:54:20.204688 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:54:20.204688 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:54:20.204688 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:54:20.204688 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:54:20.204688 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 13 15:54:20.204688 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 13 15:54:20.204688 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 13 15:54:20.204688 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 13 15:54:20.205611 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:54:20.205611 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:54:20.205611 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:54:20.205611 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:54:20.205611 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:54:20.205611 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:54:20.205611 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:54:20.205611 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application with Lock( Sep 13 15:54:20.205611 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:54:20.205611 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:54:20.205611 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:54:20.205611 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:54:20.205611 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:54:20.205611 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:54:20.205611 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:54:20.205611 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:54:20.205611 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 13 15:54:20.206520 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 13 15:54:20.206520 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:54:20.206520 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application Sep 13 15:54:20.206520 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 335/674] 10.222.0.50 () {56 vars in 1108 bytes} [Tue Sep 13 15:54:19 2022] GET /identity/v3/domains/Default => generated 87 bytes in 245 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:54:20.206520 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9b53d14-473d-4227-9ee3-6fa758e62b84 None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:20.263130 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:54:20.310324 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9b53d14-473d-4227-9ee3-6fa758e62b84 None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:20.317641 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-026645f1-9ba1-4a57-bc45-5a396de2c622 None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:20.348443 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c41f99c0-cb05-4254-8e30-1b2c743ad436 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:20.349215 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c41f99c0-cb05-4254-8e30-1b2c743ad436 None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:20.349503 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c41f99c0-cb05-4254-8e30-1b2c743ad436 None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:20.349806 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c41f99c0-cb05-4254-8e30-1b2c743ad436 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:20.350564 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c41f99c0-cb05-4254-8e30-1b2c743ad436 None admin] RBAC: Adding query filter params (role.id=da847124d9374ac88b27968451c486ab, scope.project.id=5811e3b83a1e462e9320cad79a3ad791, user.id=7fd6b5afd67b48f9b0f0597ccd4cf4df) {{(pid=81437) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:54:20.350878 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c41f99c0-cb05-4254-8e30-1b2c743ad436 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:20.352412 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c41f99c0-cb05-4254-8e30-1b2c743ad436 None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:20.356092 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d0e5382-9b0e-4149-a845-65e72dd7f0b0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:20.357022 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d0e5382-9b0e-4149-a845-65e72dd7f0b0 None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:20.357515 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d0e5382-9b0e-4149-a845-65e72dd7f0b0 None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:20.357972 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d0e5382-9b0e-4149-a845-65e72dd7f0b0 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:20.360131 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 336/675] 10.222.0.50 () {56 vars in 1374 bytes} [Tue Sep 13 15:54:20 2022] GET /identity/v3/role_assignments?user.id=7fd6b5afd67b48f9b0f0597ccd4cf4df&scope.project.id=5811e3b83a1e462e9320cad79a3ad791&role.id=da847124d9374ac88b27968451c486ab => generated 608 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:54:20.361083 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c41f99c0-cb05-4254-8e30-1b2c743ad436 None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:20.383643 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d0e5382-9b0e-4149-a845-65e72dd7f0b0 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:20.386240 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d0e5382-9b0e-4149-a845-65e72dd7f0b0 None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:20.404404 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application [None req-2d0e5382-9b0e-4149-a845-65e72dd7f0b0 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:54:20.404404 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:54:20.404404 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:54:20.404404 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:54:20.404404 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:54:20.404404 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:54:20.404404 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:54:20.404404 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:54:20.404404 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:54:20.404404 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:54:20.404404 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:54:20.404404 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:54:20.404404 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 13 15:54:20.404404 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 13 15:54:20.404404 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 13 15:54:20.404404 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 13 15:54:20.405480 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:54:20.405480 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:54:20.405480 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:54:20.405480 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:54:20.405480 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:54:20.405480 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:54:20.405480 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:54:20.405480 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application with Lock( Sep 13 15:54:20.405480 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:54:20.405480 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:54:20.405480 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:54:20.405480 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:54:20.405480 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:54:20.405480 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:54:20.405480 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:54:20.405480 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:54:20.405480 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 13 15:54:20.406407 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 13 15:54:20.406407 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:54:20.406407 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Sep 13 15:54:20.413030 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 340/676] 10.222.0.50 () {56 vars in 1108 bytes} [Tue Sep 13 15:54:20 2022] GET /identity/v3/domains/Default => generated 87 bytes in 211 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:54:20.413030 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d0e5382-9b0e-4149-a845-65e72dd7f0b0 None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:20.417358 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:54:20.501174 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c41f99c0-cb05-4254-8e30-1b2c743ad436 None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:20.531477 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:54:20.535935 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d0e5382-9b0e-4149-a845-65e72dd7f0b0 None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:20.542778 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f258976-5b42-4991-a1a3-de28c146a2b5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:20.543438 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f258976-5b42-4991-a1a3-de28c146a2b5 None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:20.543720 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f258976-5b42-4991-a1a3-de28c146a2b5 None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:20.543997 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f258976-5b42-4991-a1a3-de28c146a2b5 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:20.544745 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f258976-5b42-4991-a1a3-de28c146a2b5 None admin] RBAC: Adding query filter params (role.id=da847124d9374ac88b27968451c486ab, scope.project.id=5811e3b83a1e462e9320cad79a3ad791, user.id=6ab5f8018f464be69d4af7683f553738) {{(pid=81437) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:54:20.545058 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f258976-5b42-4991-a1a3-de28c146a2b5 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:20.546552 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f258976-5b42-4991-a1a3-de28c146a2b5 None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:20.552838 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 337/677] 10.222.0.50 () {56 vars in 1374 bytes} [Tue Sep 13 15:54:20 2022] GET /identity/v3/role_assignments?user.id=6ab5f8018f464be69d4af7683f553738&scope.project.id=5811e3b83a1e462e9320cad79a3ad791&role.id=da847124d9374ac88b27968451c486ab => generated 608 bytes in 192 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:54:20.553663 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f258976-5b42-4991-a1a3-de28c146a2b5 None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:20.588412 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da6291f5-1a44-4900-af80-5f9508a0dd46 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:20.589044 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da6291f5-1a44-4900-af80-5f9508a0dd46 None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:20.589322 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da6291f5-1a44-4900-af80-5f9508a0dd46 None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:20.589593 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da6291f5-1a44-4900-af80-5f9508a0dd46 None admin] PATH_INFO: `/v3/domains` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:20.590251 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da6291f5-1a44-4900-af80-5f9508a0dd46 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81438) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:54:20.590551 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da6291f5-1a44-4900-af80-5f9508a0dd46 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:20.592470 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da6291f5-1a44-4900-af80-5f9508a0dd46 None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:20.608798 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 341/678] 10.222.0.50 () {56 vars in 1109 bytes} [Tue Sep 13 15:54:20 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 197 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:54:20.609991 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2231b79-7684-4b39-8e60-f4fedc1376e6 None None] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:20.610277 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2231b79-7684-4b39-8e60-f4fedc1376e6 None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:20.610549 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2231b79-7684-4b39-8e60-f4fedc1376e6 None None] PATH_INFO: `/` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:20.617226 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 342/679] 10.222.0.50 () {54 vars in 927 bytes} [Tue Sep 13 15:54:20 2022] GET /identity => generated 270 bytes in 8 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:54:20.618260 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2231b79-7684-4b39-8e60-f4fedc1376e6 None None] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:20.680478 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:54:20.685157 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f258976-5b42-4991-a1a3-de28c146a2b5 None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:20.734178 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-304438e5-71ad-44f4-a3ae-e28d78c50ccc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:20.735078 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-304438e5-71ad-44f4-a3ae-e28d78c50ccc None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:20.735499 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-304438e5-71ad-44f4-a3ae-e28d78c50ccc None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:20.736502 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-304438e5-71ad-44f4-a3ae-e28d78c50ccc None admin] PATH_INFO: `/v3/domains` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:20.737303 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-304438e5-71ad-44f4-a3ae-e28d78c50ccc None admin] RBAC: Adding query filter params (name=Default) {{(pid=81437) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:54:20.738042 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-304438e5-71ad-44f4-a3ae-e28d78c50ccc None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:20.740271 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-304438e5-71ad-44f4-a3ae-e28d78c50ccc None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:20.753837 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2231b79-7684-4b39-8e60-f4fedc1376e6 None None] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:20.757150 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 338/680] 10.222.0.50 () {56 vars in 1109 bytes} [Tue Sep 13 15:54:20 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 204 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:54:20.758577 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97e71fe8-d21d-4761-81a1-7f33e03cd066 None None] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:20.759583 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97e71fe8-d21d-4761-81a1-7f33e03cd066 None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:20.760326 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97e71fe8-d21d-4761-81a1-7f33e03cd066 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:20.779757 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.auth.core [None req-97e71fe8-d21d-4761-81a1-7f33e03cd066 None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81437) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:54:20.781274 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-97e71fe8-d21d-4761-81a1-7f33e03cd066 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:54:20.798885 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-370367f5-6405-49d4-b5b9-9a2780b1f29e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:20.799646 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-370367f5-6405-49d4-b5b9-9a2780b1f29e None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:20.800188 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-370367f5-6405-49d4-b5b9-9a2780b1f29e None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:20.800188 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-370367f5-6405-49d4-b5b9-9a2780b1f29e None admin] PATH_INFO: `/v3/projects/service` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:20.808305 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-370367f5-6405-49d4-b5b9-9a2780b1f29e None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:20.809753 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-370367f5-6405-49d4-b5b9-9a2780b1f29e None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:20.828330 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application [None req-370367f5-6405-49d4-b5b9-9a2780b1f29e None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 13 15:54:20.828330 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:54:20.828330 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:54:20.828330 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:54:20.828330 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:54:20.828330 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:54:20.828330 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:54:20.828330 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:54:20.828330 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:54:20.828330 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:54:20.828330 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:54:20.828330 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:54:20.828330 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 13 15:54:20.828330 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 13 15:54:20.828330 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 13 15:54:20.829311 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 13 15:54:20.829311 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:54:20.829311 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:54:20.829311 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:54:20.829311 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:54:20.829311 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:54:20.829311 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:54:20.829311 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:54:20.829311 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application with Lock( Sep 13 15:54:20.829311 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:54:20.829311 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:54:20.829311 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:54:20.829311 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:54:20.829311 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:54:20.829311 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:54:20.829311 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:54:20.829311 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:54:20.830247 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 13 15:54:20.830247 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 13 15:54:20.830247 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 13 15:54:20.830247 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 13 15:54:20.830247 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 13 15:54:20.830247 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 13 15:54:20.830247 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 13 15:54:20.830247 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Sep 13 15:54:20.830247 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 343/681] 10.222.0.50 () {56 vars in 1111 bytes} [Tue Sep 13 15:54:20 2022] GET /identity/v3/projects/service => generated 88 bytes in 210 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:54:20.830247 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-370367f5-6405-49d4-b5b9-9a2780b1f29e None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:20.859385 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:54:20.872273 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 339/682] 10.222.0.50 () {58 vars in 1024 bytes} [Tue Sep 13 15:54:20 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 13 15:54:20.878930 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97e71fe8-d21d-4761-81a1-7f33e03cd066 None None] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:20.972358 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-370367f5-6405-49d4-b5b9-9a2780b1f29e None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:20.976788 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:54:21.008521 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97e71fe8-d21d-4761-81a1-7f33e03cd066 None None] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:21.010197 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a90d940f-fc83-48b2-b7d3-3fd815715ac1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:21.010842 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a90d940f-fc83-48b2-b7d3-3fd815715ac1 None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:21.011165 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a90d940f-fc83-48b2-b7d3-3fd815715ac1 None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:21.011440 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a90d940f-fc83-48b2-b7d3-3fd815715ac1 None admin] PATH_INFO: `/v3/users/swift` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:21.019785 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a90d940f-fc83-48b2-b7d3-3fd815715ac1 None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:21.021236 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a90d940f-fc83-48b2-b7d3-3fd815715ac1 None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:21.034645 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application [None req-a90d940f-fc83-48b2-b7d3-3fd815715ac1 None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Sep 13 15:54:21.034645 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:54:21.034645 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:54:21.034645 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:54:21.034645 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:54:21.034645 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:54:21.034645 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:54:21.034645 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:54:21.034645 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:54:21.034645 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:54:21.034645 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:54:21.034645 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:54:21.034645 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 13 15:54:21.034645 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 13 15:54:21.034645 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 13 15:54:21.034645 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 13 15:54:21.035691 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:54:21.035691 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:54:21.035691 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 13 15:54:21.035691 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 13 15:54:21.035691 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 13 15:54:21.035691 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 13 15:54:21.035691 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:54:21.035691 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:54:21.035691 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:54:21.035691 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:54:21.035691 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:54:21.035691 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application with Lock( Sep 13 15:54:21.035691 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:54:21.035691 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:54:21.035691 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:54:21.035691 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:54:21.035691 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:54:21.035691 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:54:21.036675 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:54:21.036675 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:54:21.036675 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 13 15:54:21.036675 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 13 15:54:21.036675 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 13 15:54:21.036675 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 13 15:54:21.036675 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 13 15:54:21.036675 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 13 15:54:21.036675 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Sep 13 15:54:21.036675 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Sep 13 15:54:21.036675 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 344/683] 10.222.0.50 () {56 vars in 1096 bytes} [Tue Sep 13 15:54:20 2022] GET /identity/v3/users/swift => generated 83 bytes in 207 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:54:21.042051 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a90d940f-fc83-48b2-b7d3-3fd815715ac1 None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:21.079059 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c3b94db-88f9-4566-a79f-96c8f805fba7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:21.079669 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c3b94db-88f9-4566-a79f-96c8f805fba7 None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:21.079942 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c3b94db-88f9-4566-a79f-96c8f805fba7 None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:21.080209 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c3b94db-88f9-4566-a79f-96c8f805fba7 None admin] PATH_INFO: `/v3/projects` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:21.080899 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c3b94db-88f9-4566-a79f-96c8f805fba7 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81437) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:54:21.081198 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c3b94db-88f9-4566-a79f-96c8f805fba7 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:21.082633 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c3b94db-88f9-4566-a79f-96c8f805fba7 None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:21.105050 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 340/684] 10.222.0.50 () {56 vars in 1148 bytes} [Tue Sep 13 15:54:20 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 232 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:54:21.111145 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c3b94db-88f9-4566-a79f-96c8f805fba7 None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:21.202813 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:54:21.226213 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a90d940f-fc83-48b2-b7d3-3fd815715ac1 None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:21.259518 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c3b94db-88f9-4566-a79f-96c8f805fba7 None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:21.271990 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-849158bf-f123-4faa-8e41-ef86ff9bc95b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:21.273080 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-849158bf-f123-4faa-8e41-ef86ff9bc95b None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:21.273546 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-849158bf-f123-4faa-8e41-ef86ff9bc95b None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:21.274365 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-849158bf-f123-4faa-8e41-ef86ff9bc95b None admin] PATH_INFO: `/v3/users` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:21.276129 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-849158bf-f123-4faa-8e41-ef86ff9bc95b None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=81438) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:54:21.276652 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-849158bf-f123-4faa-8e41-ef86ff9bc95b None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:21.278822 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-849158bf-f123-4faa-8e41-ef86ff9bc95b None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:21.295295 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2ee0643-541a-4fe5-aaf1-6ad26ed647a8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:21.296102 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2ee0643-541a-4fe5-aaf1-6ad26ed647a8 None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:21.296381 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2ee0643-541a-4fe5-aaf1-6ad26ed647a8 None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:21.296651 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2ee0643-541a-4fe5-aaf1-6ad26ed647a8 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:21.297371 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2ee0643-541a-4fe5-aaf1-6ad26ed647a8 None admin] RBAC: Adding query filter params (role.id=da847124d9374ac88b27968451c486ab, scope.project.id=5811e3b83a1e462e9320cad79a3ad791, user.id=2f436f07939d4855a21b195bc7566244) {{(pid=81437) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:54:21.297678 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2ee0643-541a-4fe5-aaf1-6ad26ed647a8 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:21.300454 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2ee0643-541a-4fe5-aaf1-6ad26ed647a8 None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:21.304985 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 345/685] 10.222.0.50 () {56 vars in 1135 bytes} [Tue Sep 13 15:54:21 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 263 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 13 15:54:21.306066 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-849158bf-f123-4faa-8e41-ef86ff9bc95b None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:21.311091 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 341/686] 10.222.0.50 () {56 vars in 1374 bytes} [Tue Sep 13 15:54:21 2022] GET /identity/v3/role_assignments?user.id=2f436f07939d4855a21b195bc7566244&scope.project.id=5811e3b83a1e462e9320cad79a3ad791&role.id=da847124d9374ac88b27968451c486ab => generated 608 bytes in 200 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:54:21.314746 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2ee0643-541a-4fe5-aaf1-6ad26ed647a8 None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:21.351301 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:54:21.465346 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2ee0643-541a-4fe5-aaf1-6ad26ed647a8 None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:21.519956 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-348e27ee-343d-491b-a2dd-6a5653775333 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:21.520827 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-348e27ee-343d-491b-a2dd-6a5653775333 None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:21.521271 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-348e27ee-343d-491b-a2dd-6a5653775333 None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:21.521555 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-348e27ee-343d-491b-a2dd-6a5653775333 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:21.523716 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-849158bf-f123-4faa-8e41-ef86ff9bc95b None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:21.574468 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b922ceb3-f174-459f-b085-3b68d73b2b52 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:21.576189 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b922ceb3-f174-459f-b085-3b68d73b2b52 None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:21.577129 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b922ceb3-f174-459f-b085-3b68d73b2b52 None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:21.577583 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b922ceb3-f174-459f-b085-3b68d73b2b52 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:21.589110 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:54:21.594734 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b922ceb3-f174-459f-b085-3b68d73b2b52 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 13 15:54:21.600789 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application [None req-b922ceb3-f174-459f-b085-3b68d73b2b52 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 13 15:54:21.600789 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:54:21.600789 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:54:21.600789 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:54:21.600789 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:54:21.600789 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:54:21.600789 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:54:21.600789 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:54:21.600789 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:54:21.600789 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:54:21.600789 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:54:21.600789 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:54:21.600789 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 13 15:54:21.600789 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 13 15:54:21.600789 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 13 15:54:21.600789 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 13 15:54:21.601723 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 13 15:54:21.601723 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 13 15:54:21.601723 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 13 15:54:21.601723 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 13 15:54:21.601723 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:54:21.601723 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:54:21.601723 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:54:21.601723 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:54:21.601723 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:54:21.601723 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:54:21.601723 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:54:21.601723 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application with Lock( Sep 13 15:54:21.601723 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:54:21.601723 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:54:21.601723 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:54:21.601723 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:54:21.602574 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:54:21.602574 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:54:21.602574 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:54:21.602574 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:54:21.602574 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 13 15:54:21.602574 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 13 15:54:21.602574 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 13 15:54:21.602574 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 13 15:54:21.602574 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 13 15:54:21.602574 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 13 15:54:21.602574 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 13 15:54:21.602574 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Sep 13 15:54:21.608933 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 346/687] 10.222.0.50 () {56 vars in 1096 bytes} [Tue Sep 13 15:54:21 2022] GET /identity/v3/roles/admin => generated 83 bytes in 299 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:54:21.609977 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b922ceb3-f174-459f-b085-3b68d73b2b52 None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:21.626256 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-348e27ee-343d-491b-a2dd-6a5653775333 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:21.627764 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-348e27ee-343d-491b-a2dd-6a5653775333 None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:21.646226 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:54:21.780023 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b922ceb3-f174-459f-b085-3b68d73b2b52 None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:21.831827 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1639e97-d826-4d38-9281-87f4aaf5b323 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:21.832943 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1639e97-d826-4d38-9281-87f4aaf5b323 None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:21.833461 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1639e97-d826-4d38-9281-87f4aaf5b323 None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:21.833933 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1639e97-d826-4d38-9281-87f4aaf5b323 None admin] PATH_INFO: `/v3/roles` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:21.834821 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1639e97-d826-4d38-9281-87f4aaf5b323 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81438) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:54:21.835344 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1639e97-d826-4d38-9281-87f4aaf5b323 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:21.837244 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1639e97-d826-4d38-9281-87f4aaf5b323 None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:21.838705 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 342/688] 10.222.0.50 () {58 vars in 1282 bytes} [Tue Sep 13 15:54:21 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 524 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 13 15:54:21.843943 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-348e27ee-343d-491b-a2dd-6a5653775333 None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:21.847505 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 347/689] 10.222.0.50 () {56 vars in 1099 bytes} [Tue Sep 13 15:54:21 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 238 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 13 15:54:21.853970 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1639e97-d826-4d38-9281-87f4aaf5b323 None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:21.920307 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:54:21.982691 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1639e97-d826-4d38-9281-87f4aaf5b323 None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:22.021955 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-348e27ee-343d-491b-a2dd-6a5653775333 None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:22.038633 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:54:22.067159 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2ea3c02-9dbe-4c38-8743-5fd5e7ccbd1c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:22.068802 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2ea3c02-9dbe-4c38-8743-5fd5e7ccbd1c None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:22.069816 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2ea3c02-9dbe-4c38-8743-5fd5e7ccbd1c None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:22.070625 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81e5dd96-5800-43b4-a899-94abbefc2bed None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:22.071147 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2ea3c02-9dbe-4c38-8743-5fd5e7ccbd1c None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:22.072081 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81e5dd96-5800-43b4-a899-94abbefc2bed None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:22.074029 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81e5dd96-5800-43b4-a899-94abbefc2bed None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:22.074540 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81e5dd96-5800-43b4-a899-94abbefc2bed None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:22.086684 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2ea3c02-9dbe-4c38-8743-5fd5e7ccbd1c None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:22.089688 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2ea3c02-9dbe-4c38-8743-5fd5e7ccbd1c None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:22.099724 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81e5dd96-5800-43b4-a899-94abbefc2bed None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:22.101179 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81e5dd96-5800-43b4-a899-94abbefc2bed None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:22.107223 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application [None req-b2ea3c02-9dbe-4c38-8743-5fd5e7ccbd1c None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:54:22.107223 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:54:22.107223 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:54:22.107223 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:54:22.107223 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:54:22.107223 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:54:22.107223 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:54:22.107223 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:54:22.107223 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:54:22.107223 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:54:22.107223 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:54:22.107223 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:54:22.107223 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 13 15:54:22.107223 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 13 15:54:22.107223 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 13 15:54:22.107223 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 13 15:54:22.108163 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:54:22.108163 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:54:22.108163 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:54:22.108163 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:54:22.108163 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:54:22.108163 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:54:22.108163 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:54:22.108163 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application with Lock( Sep 13 15:54:22.108163 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:54:22.108163 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:54:22.108163 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:54:22.108163 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:54:22.108163 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:54:22.108163 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:54:22.108163 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:54:22.108163 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:54:22.108163 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 13 15:54:22.109062 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 13 15:54:22.109062 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:54:22.109062 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application Sep 13 15:54:22.111241 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 343/690] 10.222.0.50 () {56 vars in 1108 bytes} [Tue Sep 13 15:54:21 2022] GET /identity/v3/domains/Default => generated 87 bytes in 268 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 13 15:54:22.119913 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2ea3c02-9dbe-4c38-8743-5fd5e7ccbd1c None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:22.130920 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application [None req-81e5dd96-5800-43b4-a899-94abbefc2bed None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:54:22.130920 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:54:22.130920 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:54:22.130920 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:54:22.130920 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:54:22.130920 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:54:22.130920 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:54:22.130920 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:54:22.130920 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:54:22.130920 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:54:22.130920 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:54:22.130920 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:54:22.130920 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 13 15:54:22.130920 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 13 15:54:22.130920 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 13 15:54:22.130920 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 13 15:54:22.131874 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:54:22.131874 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:54:22.131874 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:54:22.131874 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:54:22.131874 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:54:22.131874 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:54:22.131874 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:54:22.131874 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application with Lock( Sep 13 15:54:22.131874 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:54:22.131874 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:54:22.131874 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:54:22.131874 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:54:22.131874 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:54:22.131874 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:54:22.131874 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:54:22.131874 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:54:22.131874 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 13 15:54:22.132879 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 13 15:54:22.132879 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:54:22.132879 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Sep 13 15:54:22.134691 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 348/691] 10.222.0.50 () {56 vars in 1108 bytes} [Tue Sep 13 15:54:21 2022] GET /identity/v3/domains/Default => generated 87 bytes in 277 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 13 15:54:22.135953 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81e5dd96-5800-43b4-a899-94abbefc2bed None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:22.247195 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81e5dd96-5800-43b4-a899-94abbefc2bed None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:22.261585 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:54:22.285323 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46657afb-8bee-4d3b-9ff9-e11898b38040 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:22.286011 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46657afb-8bee-4d3b-9ff9-e11898b38040 None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:22.286301 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46657afb-8bee-4d3b-9ff9-e11898b38040 None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:22.286577 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46657afb-8bee-4d3b-9ff9-e11898b38040 None admin] PATH_INFO: `/v3/domains` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:22.287262 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46657afb-8bee-4d3b-9ff9-e11898b38040 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81438) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:54:22.287572 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46657afb-8bee-4d3b-9ff9-e11898b38040 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:22.289088 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46657afb-8bee-4d3b-9ff9-e11898b38040 None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:22.306395 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 349/692] 10.222.0.50 () {56 vars in 1109 bytes} [Tue Sep 13 15:54:22 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:54:22.308502 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46657afb-8bee-4d3b-9ff9-e11898b38040 None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:22.357394 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2ea3c02-9dbe-4c38-8743-5fd5e7ccbd1c None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:22.395533 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b59a3af9-2d5b-4185-a3dc-537e29c0e0bc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:22.396144 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b59a3af9-2d5b-4185-a3dc-537e29c0e0bc None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:22.396424 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b59a3af9-2d5b-4185-a3dc-537e29c0e0bc None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:22.396695 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b59a3af9-2d5b-4185-a3dc-537e29c0e0bc None admin] PATH_INFO: `/v3/domains` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:22.397337 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b59a3af9-2d5b-4185-a3dc-537e29c0e0bc None admin] RBAC: Adding query filter params (name=Default) {{(pid=81437) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:54:22.397639 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b59a3af9-2d5b-4185-a3dc-537e29c0e0bc None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:22.399056 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b59a3af9-2d5b-4185-a3dc-537e29c0e0bc None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:22.421379 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 344/693] 10.222.0.50 () {56 vars in 1109 bytes} [Tue Sep 13 15:54:22 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 302 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:54:22.426383 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b59a3af9-2d5b-4185-a3dc-537e29c0e0bc None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:22.426848 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46657afb-8bee-4d3b-9ff9-e11898b38040 None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:22.465865 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:54:22.478773 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e147c4e9-b1b5-45f0-9e43-eda4d976db21 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:22.479772 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e147c4e9-b1b5-45f0-9e43-eda4d976db21 None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:22.480256 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e147c4e9-b1b5-45f0-9e43-eda4d976db21 None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:22.480753 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e147c4e9-b1b5-45f0-9e43-eda4d976db21 None admin] PATH_INFO: `/v3/users/nova` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:22.488878 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e147c4e9-b1b5-45f0-9e43-eda4d976db21 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:22.490562 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e147c4e9-b1b5-45f0-9e43-eda4d976db21 None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:22.503295 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application [None req-e147c4e9-b1b5-45f0-9e43-eda4d976db21 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Sep 13 15:54:22.503295 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:54:22.503295 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:54:22.503295 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:54:22.503295 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:54:22.503295 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:54:22.503295 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:54:22.503295 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:54:22.503295 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:54:22.503295 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:54:22.503295 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:54:22.503295 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:54:22.503295 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 13 15:54:22.503295 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 13 15:54:22.503295 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 13 15:54:22.503295 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 13 15:54:22.504294 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:54:22.504294 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:54:22.504294 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 13 15:54:22.504294 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 13 15:54:22.504294 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 13 15:54:22.504294 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 13 15:54:22.504294 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:54:22.504294 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:54:22.504294 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:54:22.504294 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:54:22.504294 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:54:22.504294 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application with Lock( Sep 13 15:54:22.504294 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:54:22.504294 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:54:22.504294 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:54:22.504294 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:54:22.504294 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:54:22.504294 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:54:22.505267 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:54:22.505267 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:54:22.505267 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 13 15:54:22.505267 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 13 15:54:22.505267 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 13 15:54:22.505267 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 13 15:54:22.505267 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 13 15:54:22.505267 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 13 15:54:22.505267 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Sep 13 15:54:22.505267 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Sep 13 15:54:22.507835 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 350/694] 10.222.0.50 () {56 vars in 1093 bytes} [Tue Sep 13 15:54:22 2022] GET /identity/v3/users/nova => generated 82 bytes in 200 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:54:22.520101 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e147c4e9-b1b5-45f0-9e43-eda4d976db21 None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:22.585929 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b59a3af9-2d5b-4185-a3dc-537e29c0e0bc None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:22.603263 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:54:22.645908 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5092d365-6015-41d9-9748-4c39a7793428 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:22.646549 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5092d365-6015-41d9-9748-4c39a7793428 None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:22.646829 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5092d365-6015-41d9-9748-4c39a7793428 None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:22.647140 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5092d365-6015-41d9-9748-4c39a7793428 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:22.658514 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5092d365-6015-41d9-9748-4c39a7793428 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:22.661379 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5092d365-6015-41d9-9748-4c39a7793428 None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:22.685233 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application [None req-5092d365-6015-41d9-9748-4c39a7793428 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 13 15:54:22.685233 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:54:22.685233 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:54:22.685233 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:54:22.685233 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:54:22.685233 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:54:22.685233 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:54:22.685233 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:54:22.685233 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:54:22.685233 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:54:22.685233 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:54:22.685233 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:54:22.685233 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 13 15:54:22.685233 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 13 15:54:22.685233 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 13 15:54:22.686249 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 13 15:54:22.686249 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:54:22.686249 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:54:22.686249 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:54:22.686249 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:54:22.686249 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:54:22.686249 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:54:22.686249 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:54:22.686249 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application with Lock( Sep 13 15:54:22.686249 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:54:22.686249 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:54:22.686249 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:54:22.686249 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:54:22.686249 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:54:22.686249 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:54:22.686249 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:54:22.686249 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:54:22.687219 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 13 15:54:22.687219 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 13 15:54:22.687219 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 13 15:54:22.687219 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 13 15:54:22.687219 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 13 15:54:22.687219 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 13 15:54:22.687219 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 13 15:54:22.687219 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application Sep 13 15:54:22.693006 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 345/695] 10.222.0.50 () {56 vars in 1111 bytes} [Tue Sep 13 15:54:22 2022] GET /identity/v3/projects/service => generated 88 bytes in 267 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:54:22.694044 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5092d365-6015-41d9-9748-4c39a7793428 None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:22.769067 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e147c4e9-b1b5-45f0-9e43-eda4d976db21 None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:22.799999 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:54:22.819840 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5092d365-6015-41d9-9748-4c39a7793428 None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:22.862276 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f333e054-4051-49f7-95ca-c263b00d57bf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:22.862833 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f333e054-4051-49f7-95ca-c263b00d57bf None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:22.863168 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f333e054-4051-49f7-95ca-c263b00d57bf None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:22.863442 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f333e054-4051-49f7-95ca-c263b00d57bf None admin] PATH_INFO: `/v3/projects` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:22.864209 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f333e054-4051-49f7-95ca-c263b00d57bf None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81437) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:54:22.864524 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f333e054-4051-49f7-95ca-c263b00d57bf None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:22.866076 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f333e054-4051-49f7-95ca-c263b00d57bf None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:22.875783 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4286f777-98f7-4c68-975a-b895b4afeb2d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:22.876466 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4286f777-98f7-4c68-975a-b895b4afeb2d None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:22.876690 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4286f777-98f7-4c68-975a-b895b4afeb2d None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:22.876965 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4286f777-98f7-4c68-975a-b895b4afeb2d None admin] PATH_INFO: `/v3/users` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:22.877707 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4286f777-98f7-4c68-975a-b895b4afeb2d None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81438) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:54:22.878011 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4286f777-98f7-4c68-975a-b895b4afeb2d None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:22.879618 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4286f777-98f7-4c68-975a-b895b4afeb2d None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:22.886552 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 346/696] 10.222.0.50 () {56 vars in 1148 bytes} [Tue Sep 13 15:54:22 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 188 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:54:22.892408 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f333e054-4051-49f7-95ca-c263b00d57bf None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:22.912347 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 351/697] 10.222.0.50 () {56 vars in 1133 bytes} [Tue Sep 13 15:54:22 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 393 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:54:22.920132 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4286f777-98f7-4c68-975a-b895b4afeb2d None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:23.027925 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:54:23.039640 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4286f777-98f7-4c68-975a-b895b4afeb2d None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:23.046222 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f333e054-4051-49f7-95ca-c263b00d57bf None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:23.087516 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aeb55e1a-f69a-4ada-a12c-7e9d2928cb2e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:23.088584 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aeb55e1a-f69a-4ada-a12c-7e9d2928cb2e None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:23.089008 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6dd38b3-58c6-49bc-8025-892f258ab67d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:23.089304 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aeb55e1a-f69a-4ada-a12c-7e9d2928cb2e None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:23.089617 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6dd38b3-58c6-49bc-8025-892f258ab67d None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:23.089896 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6dd38b3-58c6-49bc-8025-892f258ab67d None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:23.090171 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6dd38b3-58c6-49bc-8025-892f258ab67d None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:23.090435 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aeb55e1a-f69a-4ada-a12c-7e9d2928cb2e None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:23.090893 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6dd38b3-58c6-49bc-8025-892f258ab67d None admin] RBAC: Adding query filter params (role.id=da847124d9374ac88b27968451c486ab, scope.project.id=5811e3b83a1e462e9320cad79a3ad791, user.id=b00ab0d6321d483188bc69deaddcdbbc) {{(pid=81437) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:54:23.091227 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6dd38b3-58c6-49bc-8025-892f258ab67d None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:23.092670 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6dd38b3-58c6-49bc-8025-892f258ab67d None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:23.103436 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 347/698] 10.222.0.50 () {56 vars in 1374 bytes} [Tue Sep 13 15:54:22 2022] GET /identity/v3/role_assignments?user.id=b00ab0d6321d483188bc69deaddcdbbc&scope.project.id=5811e3b83a1e462e9320cad79a3ad791&role.id=da847124d9374ac88b27968451c486ab => generated 608 bytes in 211 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 13 15:54:23.106192 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca7dd45e-95c8-46ad-bb0d-69d882dffa33 None None] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:23.106476 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca7dd45e-95c8-46ad-bb0d-69d882dffa33 None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:23.106748 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca7dd45e-95c8-46ad-bb0d-69d882dffa33 None None] PATH_INFO: `/` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:23.108278 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 348/699] 10.222.0.50 () {54 vars in 927 bytes} [Tue Sep 13 15:54:23 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:54:23.119014 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a90687b-0403-4306-9ddd-f7c2665f980b None None] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:23.119014 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a90687b-0403-4306-9ddd-f7c2665f980b None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:23.119014 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a90687b-0403-4306-9ddd-f7c2665f980b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:23.150374 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.auth.core [None req-0a90687b-0403-4306-9ddd-f7c2665f980b None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81437) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:54:23.151639 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-0a90687b-0403-4306-9ddd-f7c2665f980b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:54:23.154260 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:54:23.251904 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aeb55e1a-f69a-4ada-a12c-7e9d2928cb2e None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:23.253369 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aeb55e1a-f69a-4ada-a12c-7e9d2928cb2e None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:23.288343 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 349/700] 10.222.0.50 () {58 vars in 1024 bytes} [Tue Sep 13 15:54:23 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 175 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 13 15:54:23.299023 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-964dceb1-f3bc-4893-8fe0-fd74e5a4f4d2 None None] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:23.299311 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-964dceb1-f3bc-4893-8fe0-fd74e5a4f4d2 None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:23.299587 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-964dceb1-f3bc-4893-8fe0-fd74e5a4f4d2 None None] PATH_INFO: `/` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:23.301149 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 350/701] 10.222.0.50 () {54 vars in 927 bytes} [Tue Sep 13 15:54:23 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 13 15:54:23.308156 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa532ea3-e2c2-433d-810d-22cc810b112e None None] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:23.308433 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa532ea3-e2c2-433d-810d-22cc810b112e None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:23.308707 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa532ea3-e2c2-433d-810d-22cc810b112e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:23.332321 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.auth.core [None req-aa532ea3-e2c2-433d-810d-22cc810b112e None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81437) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:54:23.333520 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-aa532ea3-e2c2-433d-810d-22cc810b112e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:54:23.386836 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:54:23.398723 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 352/702] 10.222.0.50 () {58 vars in 1282 bytes} [Tue Sep 13 15:54:22 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 479 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 13 15:54:23.403985 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aeb55e1a-f69a-4ada-a12c-7e9d2928cb2e None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:23.408058 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 351/703] 10.222.0.50 () {58 vars in 1024 bytes} [Tue Sep 13 15:54:23 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 13 15:54:23.508643 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aeb55e1a-f69a-4ada-a12c-7e9d2928cb2e None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:23.544967 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea5ad5f1-dba7-4dd7-9f7d-2450832ebc08 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:23.545586 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea5ad5f1-dba7-4dd7-9f7d-2450832ebc08 None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:23.545867 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea5ad5f1-dba7-4dd7-9f7d-2450832ebc08 None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:23.546138 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea5ad5f1-dba7-4dd7-9f7d-2450832ebc08 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:23.557359 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea5ad5f1-dba7-4dd7-9f7d-2450832ebc08 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:23.559107 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea5ad5f1-dba7-4dd7-9f7d-2450832ebc08 None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:23.575355 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application [None req-ea5ad5f1-dba7-4dd7-9f7d-2450832ebc08 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:54:23.575355 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:54:23.575355 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:54:23.575355 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:54:23.575355 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:54:23.575355 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:54:23.575355 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:54:23.575355 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:54:23.575355 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:54:23.575355 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:54:23.575355 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:54:23.575355 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:54:23.575355 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 13 15:54:23.575355 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 13 15:54:23.575355 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 13 15:54:23.575355 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 13 15:54:23.576457 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:54:23.576457 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:54:23.576457 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:54:23.576457 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:54:23.576457 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:54:23.576457 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:54:23.576457 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:54:23.576457 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application with Lock( Sep 13 15:54:23.576457 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:54:23.576457 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:54:23.576457 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:54:23.576457 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:54:23.576457 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:54:23.576457 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:54:23.576457 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:54:23.576457 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:54:23.576457 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 13 15:54:23.577370 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 13 15:54:23.577370 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:54:23.577370 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Sep 13 15:54:23.577370 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 353/704] 10.222.0.50 () {56 vars in 1108 bytes} [Tue Sep 13 15:54:23 2022] GET /identity/v3/domains/Default => generated 87 bytes in 173 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:54:23.581888 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa532ea3-e2c2-433d-810d-22cc810b112e None None] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:23.600692 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea5ad5f1-dba7-4dd7-9f7d-2450832ebc08 None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:23.651866 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:54:23.660206 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:54:23.698941 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa532ea3-e2c2-433d-810d-22cc810b112e None None] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:23.718951 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea5ad5f1-dba7-4dd7-9f7d-2450832ebc08 None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:23.741478 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2db10da9-1b1c-4956-b185-c9d7349a12a3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:23.742234 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2db10da9-1b1c-4956-b185-c9d7349a12a3 None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:23.742516 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2db10da9-1b1c-4956-b185-c9d7349a12a3 None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:23.742988 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2db10da9-1b1c-4956-b185-c9d7349a12a3 None admin] PATH_INFO: `/v3/domains` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:23.744345 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2db10da9-1b1c-4956-b185-c9d7349a12a3 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81437) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:54:23.744931 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2db10da9-1b1c-4956-b185-c9d7349a12a3 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:23.746551 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2db10da9-1b1c-4956-b185-c9d7349a12a3 None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:23.754602 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-faa7905b-06f8-4085-80a7-8a0a7e5cab9f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:23.755284 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faa7905b-06f8-4085-80a7-8a0a7e5cab9f None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:23.755564 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faa7905b-06f8-4085-80a7-8a0a7e5cab9f None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:23.755836 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faa7905b-06f8-4085-80a7-8a0a7e5cab9f None admin] PATH_INFO: `/v3/services/block-storage` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:23.756417 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-faa7905b-06f8-4085-80a7-8a0a7e5cab9f None admin] RBAC: Authorizing `identity:get_service(service_id=block-storage)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:23.757835 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-faa7905b-06f8-4085-80a7-8a0a7e5cab9f None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:23.760606 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 352/705] 10.222.0.50 () {56 vars in 1109 bytes} [Tue Sep 13 15:54:23 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:54:23.761537 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2db10da9-1b1c-4956-b185-c9d7349a12a3 None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:23.769708 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application [None req-faa7905b-06f8-4085-80a7-8a0a7e5cab9f None admin] Could not find service: block-storage.: keystone.exception.ServiceNotFound: Could not find service: block-storage. Sep 13 15:54:23.769708 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:54:23.769708 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 13 15:54:23.769708 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 13 15:54:23.769708 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 13 15:54:23.769708 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 13 15:54:23.769708 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 13 15:54:23.769708 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 13 15:54:23.769708 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Sep 13 15:54:23.769708 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Sep 13 15:54:23.769708 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 13 15:54:23.769708 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Sep 13 15:54:23.769708 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:54:23.769708 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:54:23.769708 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:54:23.769708 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:54:23.769708 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:54:23.770788 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:54:23.770788 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:54:23.770788 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:54:23.770788 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:54:23.770788 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:54:23.770788 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:54:23.770788 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 13 15:54:23.770788 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 13 15:54:23.770788 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 13 15:54:23.770788 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 13 15:54:23.770788 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:54:23.770788 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:54:23.770788 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:54:23.770788 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:54:23.770788 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:54:23.770788 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:54:23.771691 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:54:23.771691 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application with Lock( Sep 13 15:54:23.771691 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:54:23.771691 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:54:23.771691 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:54:23.771691 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:54:23.771691 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:54:23.771691 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:54:23.771691 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:54:23.771691 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:54:23.771691 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 13 15:54:23.771691 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 13 15:54:23.771691 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Sep 13 15:54:23.771691 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Sep 13 15:54:23.771691 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 354/706] 10.222.0.50 () {56 vars in 1129 bytes} [Tue Sep 13 15:54:23 2022] GET /identity/v3/services/block-storage => generated 94 bytes in 171 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 13 15:54:23.773574 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-faa7905b-06f8-4085-80a7-8a0a7e5cab9f None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:23.888970 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-faa7905b-06f8-4085-80a7-8a0a7e5cab9f None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:23.892268 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2db10da9-1b1c-4956-b185-c9d7349a12a3 None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:23.898299 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:54:23.926715 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:54:23.937122 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79235822-13ce-431f-9169-0242bfea6caa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:23.937399 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88072d3e-9970-4787-8282-f8a27de059d5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:23.937783 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79235822-13ce-431f-9169-0242bfea6caa None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:23.938063 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79235822-13ce-431f-9169-0242bfea6caa None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:23.938333 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88072d3e-9970-4787-8282-f8a27de059d5 None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:23.938445 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79235822-13ce-431f-9169-0242bfea6caa None admin] PATH_INFO: `/v3/projects/service` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:23.938808 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88072d3e-9970-4787-8282-f8a27de059d5 None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:23.939124 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88072d3e-9970-4787-8282-f8a27de059d5 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:23.947597 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79235822-13ce-431f-9169-0242bfea6caa None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:23.949356 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79235822-13ce-431f-9169-0242bfea6caa None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:23.952410 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88072d3e-9970-4787-8282-f8a27de059d5 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:23.954170 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88072d3e-9970-4787-8282-f8a27de059d5 None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:23.965361 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application [None req-79235822-13ce-431f-9169-0242bfea6caa None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 13 15:54:23.965361 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:54:23.965361 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:54:23.965361 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:54:23.965361 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:54:23.965361 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:54:23.965361 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:54:23.965361 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:54:23.965361 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:54:23.965361 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:54:23.965361 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:54:23.965361 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:54:23.965361 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 13 15:54:23.965361 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 13 15:54:23.965361 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 13 15:54:23.967279 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 13 15:54:23.967279 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:54:23.967279 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:54:23.967279 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:54:23.967279 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:54:23.967279 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:54:23.967279 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:54:23.967279 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:54:23.967279 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application with Lock( Sep 13 15:54:23.967279 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:54:23.967279 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:54:23.967279 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:54:23.967279 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:54:23.967279 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:54:23.967279 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:54:23.967279 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:54:23.967279 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:54:23.969242 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 13 15:54:23.969242 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 13 15:54:23.969242 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 13 15:54:23.969242 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 13 15:54:23.969242 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 13 15:54:23.969242 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 13 15:54:23.969242 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 13 15:54:23.969242 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Sep 13 15:54:23.969242 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 355/707] 10.222.0.50 () {56 vars in 1111 bytes} [Tue Sep 13 15:54:23 2022] GET /identity/v3/projects/service => generated 88 bytes in 196 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:54:23.969242 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79235822-13ce-431f-9169-0242bfea6caa None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:23.979101 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application [None req-88072d3e-9970-4787-8282-f8a27de059d5 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:54:23.979101 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:54:23.979101 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:54:23.979101 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:54:23.979101 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:54:23.979101 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:54:23.979101 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:54:23.979101 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:54:23.979101 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:54:23.979101 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:54:23.979101 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:54:23.979101 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:54:23.979101 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 13 15:54:23.979101 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 13 15:54:23.979101 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 13 15:54:23.979101 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 13 15:54:23.981034 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:54:23.981034 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:54:23.981034 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:54:23.981034 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:54:23.981034 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:54:23.981034 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:54:23.981034 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:54:23.981034 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application with Lock( Sep 13 15:54:23.981034 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:54:23.981034 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:54:23.981034 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:54:23.981034 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:54:23.981034 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:54:23.981034 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:54:23.981034 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:54:23.981034 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:54:23.981034 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 13 15:54:23.982781 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 13 15:54:23.982781 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:54:23.982781 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application Sep 13 15:54:23.982781 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 353/708] 10.222.0.50 () {56 vars in 1108 bytes} [Tue Sep 13 15:54:23 2022] GET /identity/v3/domains/Default => generated 87 bytes in 214 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:54:23.982781 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88072d3e-9970-4787-8282-f8a27de059d5 None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:24.088422 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88072d3e-9970-4787-8282-f8a27de059d5 None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:24.089026 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79235822-13ce-431f-9169-0242bfea6caa None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:24.122534 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8383e9c-b603-4594-bc27-9390b3b17801 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:24.123169 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8383e9c-b603-4594-bc27-9390b3b17801 None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:24.123444 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8383e9c-b603-4594-bc27-9390b3b17801 None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:24.123717 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8383e9c-b603-4594-bc27-9390b3b17801 None admin] PATH_INFO: `/v3/projects` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:24.124399 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8383e9c-b603-4594-bc27-9390b3b17801 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81437) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:54:24.124705 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8383e9c-b603-4594-bc27-9390b3b17801 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:24.126154 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8383e9c-b603-4594-bc27-9390b3b17801 None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:24.126337 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cea66217-c0a6-4ca4-93e4-e4c4f3d52030 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:24.126940 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cea66217-c0a6-4ca4-93e4-e4c4f3d52030 None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:24.127226 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cea66217-c0a6-4ca4-93e4-e4c4f3d52030 None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:24.127675 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cea66217-c0a6-4ca4-93e4-e4c4f3d52030 None admin] PATH_INFO: `/v3/services` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:24.128294 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cea66217-c0a6-4ca4-93e4-e4c4f3d52030 None admin] RBAC: Adding query filter params (name=block-storage) {{(pid=81438) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:54:24.128752 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cea66217-c0a6-4ca4-93e4-e4c4f3d52030 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:24.130126 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cea66217-c0a6-4ca4-93e4-e4c4f3d52030 None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:24.139489 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 356/709] 10.222.0.50 () {56 vars in 1124 bytes} [Tue Sep 13 15:54:23 2022] GET /identity/v3/services?name=block-storage => generated 130 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:54:24.139489 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cea66217-c0a6-4ca4-93e4-e4c4f3d52030 None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:24.143292 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 354/710] 10.222.0.50 () {56 vars in 1148 bytes} [Tue Sep 13 15:54:23 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:54:24.150249 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bf00daa-0f03-4488-a30b-55b013762fa3 None None] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:24.150249 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bf00daa-0f03-4488-a30b-55b013762fa3 None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:24.150249 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bf00daa-0f03-4488-a30b-55b013762fa3 None None] PATH_INFO: `/` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:24.150249 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 355/711] 10.222.0.50 () {54 vars in 927 bytes} [Tue Sep 13 15:54:24 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:54:24.150249 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bf00daa-0f03-4488-a30b-55b013762fa3 None None] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:24.176997 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:54:24.195362 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:54:24.256352 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cea66217-c0a6-4ca4-93e4-e4c4f3d52030 None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:24.257305 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bf00daa-0f03-4488-a30b-55b013762fa3 None None] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:24.293765 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37b5e05f-2a9b-437a-bc2f-023662ce59a8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:24.294378 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37b5e05f-2a9b-437a-bc2f-023662ce59a8 None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:24.294659 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37b5e05f-2a9b-437a-bc2f-023662ce59a8 None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:24.294832 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2d53d4f-7e56-4670-88ed-4e608548c37f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:24.294969 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37b5e05f-2a9b-437a-bc2f-023662ce59a8 None admin] PATH_INFO: `/v3/domains` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:24.295423 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2d53d4f-7e56-4670-88ed-4e608548c37f None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:24.295601 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37b5e05f-2a9b-437a-bc2f-023662ce59a8 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81438) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:54:24.295785 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2d53d4f-7e56-4670-88ed-4e608548c37f None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:24.295936 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37b5e05f-2a9b-437a-bc2f-023662ce59a8 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:24.296029 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2d53d4f-7e56-4670-88ed-4e608548c37f None admin] PATH_INFO: `/v3/services` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:24.296578 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2d53d4f-7e56-4670-88ed-4e608548c37f None admin] RBAC: Adding query filter params (type=block-storage) {{(pid=81437) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:54:24.296883 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2d53d4f-7e56-4670-88ed-4e608548c37f None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:24.297242 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37b5e05f-2a9b-437a-bc2f-023662ce59a8 None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:24.298257 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2d53d4f-7e56-4670-88ed-4e608548c37f None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:24.304071 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 356/712] 10.222.0.50 () {56 vars in 1124 bytes} [Tue Sep 13 15:54:24 2022] GET /identity/v3/services?type=block-storage => generated 130 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:54:24.308757 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2d53d4f-7e56-4670-88ed-4e608548c37f None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:24.316080 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 357/713] 10.222.0.50 () {56 vars in 1109 bytes} [Tue Sep 13 15:54:24 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 180 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:54:24.317714 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebf49338-1ea2-4d85-8233-4a52314e04fa None None] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:24.318117 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebf49338-1ea2-4d85-8233-4a52314e04fa None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:24.318506 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebf49338-1ea2-4d85-8233-4a52314e04fa None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:24.339127 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.auth.core [None req-ebf49338-1ea2-4d85-8233-4a52314e04fa None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81438) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:54:24.340541 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-ebf49338-1ea2-4d85-8233-4a52314e04fa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:54:24.439532 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 358/714] 10.222.0.50 () {58 vars in 1024 bytes} [Tue Sep 13 15:54:24 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 123 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 13 15:54:24.440709 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebf49338-1ea2-4d85-8233-4a52314e04fa None None] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:24.451890 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2d53d4f-7e56-4670-88ed-4e608548c37f None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:24.456140 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:54:24.496372 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:54:24.506235 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7355d52e-c8bf-45cc-b80f-57564bc37ed8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:24.507018 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7355d52e-c8bf-45cc-b80f-57564bc37ed8 None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:24.507304 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7355d52e-c8bf-45cc-b80f-57564bc37ed8 None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:24.507582 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7355d52e-c8bf-45cc-b80f-57564bc37ed8 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:24.508312 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7355d52e-c8bf-45cc-b80f-57564bc37ed8 None admin] RBAC: Adding query filter params (role.id=031d4e1a23e34f71858fc4a6dfbd4ccb, scope.project.id=5811e3b83a1e462e9320cad79a3ad791, user.id=4e039b46c5e3445bbb39904b00c5430e) {{(pid=81437) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:54:24.508622 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7355d52e-c8bf-45cc-b80f-57564bc37ed8 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:24.510871 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7355d52e-c8bf-45cc-b80f-57564bc37ed8 None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:24.517938 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 357/715] 10.222.0.50 () {56 vars in 1374 bytes} [Tue Sep 13 15:54:24 2022] GET /identity/v3/role_assignments?user.id=4e039b46c5e3445bbb39904b00c5430e&scope.project.id=5811e3b83a1e462e9320cad79a3ad791&role.id=031d4e1a23e34f71858fc4a6dfbd4ccb => generated 259 bytes in 213 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:54:24.588482 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebf49338-1ea2-4d85-8233-4a52314e04fa None None] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:24.637420 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b95d9a94-8d1c-4d2e-8a0f-d865a96678e5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:24.638039 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b95d9a94-8d1c-4d2e-8a0f-d865a96678e5 None admin] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:24.638314 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b95d9a94-8d1c-4d2e-8a0f-d865a96678e5 None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:24.638584 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b95d9a94-8d1c-4d2e-8a0f-d865a96678e5 None admin] PATH_INFO: `/v3/users` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:24.639266 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b95d9a94-8d1c-4d2e-8a0f-d865a96678e5 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:24.640795 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b95d9a94-8d1c-4d2e-8a0f-d865a96678e5 None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:24.671086 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 359/716] 10.222.0.50 () {60 vars in 1132 bytes} [Tue Sep 13 15:54:24 2022] POST /identity/v3/users => generated 254 bytes in 226 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 15:54:24.869693 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7355d52e-c8bf-45cc-b80f-57564bc37ed8 None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:24.981212 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7355d52e-c8bf-45cc-b80f-57564bc37ed8 None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:25.016657 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93ae1822-b9b4-4a40-b02a-46a116385be7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:25.017650 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93ae1822-b9b4-4a40-b02a-46a116385be7 None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:25.018139 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93ae1822-b9b4-4a40-b02a-46a116385be7 None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:25.018643 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93ae1822-b9b4-4a40-b02a-46a116385be7 None admin] PATH_INFO: `/v3/services/network` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:25.019546 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93ae1822-b9b4-4a40-b02a-46a116385be7 None admin] RBAC: Authorizing `identity:get_service(service_id=network)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:25.021134 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93ae1822-b9b4-4a40-b02a-46a116385be7 None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:25.033303 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application [None req-93ae1822-b9b4-4a40-b02a-46a116385be7 None admin] Could not find service: network.: keystone.exception.ServiceNotFound: Could not find service: network. Sep 13 15:54:25.033303 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:54:25.033303 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 13 15:54:25.033303 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 13 15:54:25.033303 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 13 15:54:25.033303 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 13 15:54:25.033303 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 13 15:54:25.033303 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 13 15:54:25.033303 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Sep 13 15:54:25.033303 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application Sep 13 15:54:25.033303 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 13 15:54:25.033303 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application Sep 13 15:54:25.033303 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:54:25.033303 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:54:25.033303 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:54:25.033303 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:54:25.033303 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:54:25.034375 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:54:25.034375 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:54:25.034375 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:54:25.034375 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:54:25.034375 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:54:25.034375 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:54:25.034375 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 13 15:54:25.034375 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 13 15:54:25.034375 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 13 15:54:25.034375 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 13 15:54:25.034375 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:54:25.034375 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:54:25.034375 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:54:25.034375 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:54:25.034375 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:54:25.034375 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:54:25.035278 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:54:25.035278 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application with Lock( Sep 13 15:54:25.035278 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:54:25.035278 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:54:25.035278 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:54:25.035278 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:54:25.035278 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:54:25.035278 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:54:25.035278 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:54:25.035278 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:54:25.035278 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 13 15:54:25.035278 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 13 15:54:25.035278 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Sep 13 15:54:25.035278 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application Sep 13 15:54:25.039228 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 358/717] 10.222.0.50 () {56 vars in 1111 bytes} [Tue Sep 13 15:54:24 2022] GET /identity/v3/services/network => generated 88 bytes in 168 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:54:25.044480 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b95d9a94-8d1c-4d2e-8a0f-d865a96678e5 None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:25.140563 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:54:25.166213 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b95d9a94-8d1c-4d2e-8a0f-d865a96678e5 None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:25.202815 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0a31a99-4ade-4d78-a081-857769977e32 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:25.203704 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0a31a99-4ade-4d78-a081-857769977e32 None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:25.204137 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0a31a99-4ade-4d78-a081-857769977e32 None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:25.204588 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0a31a99-4ade-4d78-a081-857769977e32 None admin] PATH_INFO: `/v3/services` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:25.205416 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0a31a99-4ade-4d78-a081-857769977e32 None admin] RBAC: Adding query filter params (name=network) {{(pid=81438) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:54:25.205866 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0a31a99-4ade-4d78-a081-857769977e32 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:25.207406 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0a31a99-4ade-4d78-a081-857769977e32 None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:25.212551 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 360/718] 10.222.0.50 () {56 vars in 1112 bytes} [Tue Sep 13 15:54:25 2022] GET /identity/v3/services?name=network => generated 124 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:54:25.223556 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93ae1822-b9b4-4a40-b02a-46a116385be7 None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:25.266078 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:54:25.356815 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93ae1822-b9b4-4a40-b02a-46a116385be7 None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:25.403349 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf940525-d52a-46a8-b5a3-a36ab32103da None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:25.404198 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf940525-d52a-46a8-b5a3-a36ab32103da None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:25.404703 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf940525-d52a-46a8-b5a3-a36ab32103da None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:25.405144 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf940525-d52a-46a8-b5a3-a36ab32103da None admin] PATH_INFO: `/v3/services` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:25.405929 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf940525-d52a-46a8-b5a3-a36ab32103da None admin] RBAC: Adding query filter params (type=network) {{(pid=81437) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:54:25.406383 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf940525-d52a-46a8-b5a3-a36ab32103da None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:25.407943 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf940525-d52a-46a8-b5a3-a36ab32103da None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:25.416821 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 359/719] 10.222.0.50 () {56 vars in 1112 bytes} [Tue Sep 13 15:54:25 2022] GET /identity/v3/services?type=network => generated 124 bytes in 194 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:54:25.974555 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9446d18-08b1-492a-93ff-c6c090cd07f0 None None] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:25.974836 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9446d18-08b1-492a-93ff-c6c090cd07f0 None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:25.975139 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9446d18-08b1-492a-93ff-c6c090cd07f0 None None] PATH_INFO: `/` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:25.976694 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 361/720] 10.222.0.50 () {54 vars in 927 bytes} [Tue Sep 13 15:54:25 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:54:25.982270 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18f9cad3-9204-4e15-ba71-561ca34edee2 None None] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:25.982562 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18f9cad3-9204-4e15-ba71-561ca34edee2 None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:25.982841 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18f9cad3-9204-4e15-ba71-561ca34edee2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:26.010627 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.auth.core [None req-18f9cad3-9204-4e15-ba71-561ca34edee2 None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81437) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:54:26.011860 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-18f9cad3-9204-4e15-ba71-561ca34edee2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:54:26.120482 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 360/721] 10.222.0.50 () {58 vars in 1024 bytes} [Tue Sep 13 15:54:25 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 139 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 13 15:54:26.499474 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9446d18-08b1-492a-93ff-c6c090cd07f0 None None] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:26.642608 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9446d18-08b1-492a-93ff-c6c090cd07f0 None None] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:26.690362 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6adcdcc-8507-4f9f-9c11-cc0b24a6a93b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:26.691035 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6adcdcc-8507-4f9f-9c11-cc0b24a6a93b None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:26.691314 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6adcdcc-8507-4f9f-9c11-cc0b24a6a93b None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:26.691809 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6adcdcc-8507-4f9f-9c11-cc0b24a6a93b None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:26.705920 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:54:26.710542 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6adcdcc-8507-4f9f-9c11-cc0b24a6a93b None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 13 15:54:26.717391 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application [None req-d6adcdcc-8507-4f9f-9c11-cc0b24a6a93b None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 13 15:54:26.717391 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:54:26.717391 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:54:26.717391 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:54:26.717391 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:54:26.717391 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:54:26.717391 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:54:26.717391 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:54:26.717391 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:54:26.717391 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:54:26.717391 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:54:26.717391 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:54:26.717391 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 13 15:54:26.717391 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 13 15:54:26.717391 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 13 15:54:26.717391 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 13 15:54:26.718431 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 13 15:54:26.718431 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 13 15:54:26.718431 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 13 15:54:26.718431 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 13 15:54:26.718431 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:54:26.718431 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:54:26.718431 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:54:26.718431 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:54:26.718431 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:54:26.718431 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:54:26.718431 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:54:26.718431 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application with Lock( Sep 13 15:54:26.718431 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:54:26.718431 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:54:26.718431 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:54:26.718431 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:54:26.719886 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:54:26.719886 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:54:26.719886 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:54:26.719886 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:54:26.719886 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 13 15:54:26.719886 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 13 15:54:26.719886 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 13 15:54:26.719886 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 13 15:54:26.719886 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 13 15:54:26.719886 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 13 15:54:26.719886 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 13 15:54:26.719886 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Sep 13 15:54:26.719886 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 362/722] 10.222.0.50 () {56 vars in 1096 bytes} [Tue Sep 13 15:54:26 2022] GET /identity/v3/roles/admin => generated 83 bytes in 220 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:54:26.724720 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18f9cad3-9204-4e15-ba71-561ca34edee2 None None] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:26.763697 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:54:26.867018 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18f9cad3-9204-4e15-ba71-561ca34edee2 None None] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:26.905687 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e590c03c-5f4c-4740-93d1-98e5d40b3ba3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:26.906292 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e590c03c-5f4c-4740-93d1-98e5d40b3ba3 None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:26.906568 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e590c03c-5f4c-4740-93d1-98e5d40b3ba3 None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:26.906838 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e590c03c-5f4c-4740-93d1-98e5d40b3ba3 None admin] PATH_INFO: `/v3/roles` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:26.907524 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e590c03c-5f4c-4740-93d1-98e5d40b3ba3 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81437) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:54:26.907824 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e590c03c-5f4c-4740-93d1-98e5d40b3ba3 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:26.909175 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e590c03c-5f4c-4740-93d1-98e5d40b3ba3 None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:26.918839 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 361/723] 10.222.0.50 () {56 vars in 1099 bytes} [Tue Sep 13 15:54:26 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 194 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:54:26.924493 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6adcdcc-8507-4f9f-9c11-cc0b24a6a93b None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:26.986839 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5de5b191-8fca-4625-ab22-6cb03a117f4f None None] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:26.987200 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5de5b191-8fca-4625-ab22-6cb03a117f4f None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:26.987418 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5de5b191-8fca-4625-ab22-6cb03a117f4f None None] PATH_INFO: `/` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:26.989015 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 362/724] 10.222.0.50 () {54 vars in 927 bytes} [Tue Sep 13 15:54:26 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 13 15:54:26.995287 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-567572bd-db49-4b8b-8ce3-3204cf507e8f None None] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:26.995822 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-567572bd-db49-4b8b-8ce3-3204cf507e8f None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:26.996295 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-567572bd-db49-4b8b-8ce3-3204cf507e8f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:27.036019 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.auth.core [None req-567572bd-db49-4b8b-8ce3-3204cf507e8f None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81437) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:54:27.037537 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-567572bd-db49-4b8b-8ce3-3204cf507e8f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:54:27.067970 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6adcdcc-8507-4f9f-9c11-cc0b24a6a93b None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:27.105935 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddaa75ab-68cd-4239-bec5-dddefab56159 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:27.106892 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddaa75ab-68cd-4239-bec5-dddefab56159 None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:27.107336 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddaa75ab-68cd-4239-bec5-dddefab56159 None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:27.107768 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddaa75ab-68cd-4239-bec5-dddefab56159 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:27.120481 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ddaa75ab-68cd-4239-bec5-dddefab56159 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:27.122377 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ddaa75ab-68cd-4239-bec5-dddefab56159 None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:27.137739 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application [None req-ddaa75ab-68cd-4239-bec5-dddefab56159 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:54:27.137739 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:54:27.137739 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:54:27.137739 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:54:27.137739 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:54:27.137739 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:54:27.137739 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:54:27.137739 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:54:27.137739 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:54:27.137739 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:54:27.137739 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:54:27.137739 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:54:27.137739 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 13 15:54:27.137739 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 13 15:54:27.137739 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 13 15:54:27.137739 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 13 15:54:27.138761 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:54:27.138761 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:54:27.138761 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:54:27.138761 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:54:27.138761 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:54:27.138761 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:54:27.138761 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:54:27.138761 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application with Lock( Sep 13 15:54:27.138761 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:54:27.138761 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:54:27.138761 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:54:27.138761 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:54:27.138761 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:54:27.138761 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:54:27.138761 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:54:27.138761 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:54:27.138761 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 13 15:54:27.139688 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 13 15:54:27.139688 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:54:27.139688 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Sep 13 15:54:27.142282 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 363/725] 10.222.0.50 () {56 vars in 1108 bytes} [Tue Sep 13 15:54:26 2022] GET /identity/v3/domains/Default => generated 87 bytes in 218 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:54:27.153984 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddaa75ab-68cd-4239-bec5-dddefab56159 None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:27.218048 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:54:27.231326 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 363/726] 10.222.0.50 () {58 vars in 1024 bytes} [Tue Sep 13 15:54:26 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 237 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 13 15:54:27.270558 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddaa75ab-68cd-4239-bec5-dddefab56159 None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:27.302682 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35251cdc-53a5-451d-a997-7e5c1b397572 None None] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:27.302914 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35251cdc-53a5-451d-a997-7e5c1b397572 None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:27.303207 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35251cdc-53a5-451d-a997-7e5c1b397572 None None] PATH_INFO: `/` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:27.304726 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 364/727] 10.222.0.50 () {54 vars in 927 bytes} [Tue Sep 13 15:54:27 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:54:27.311436 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c983cdca-e02e-43f0-8ed5-9b66a31c5f72 None None] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:27.311694 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0ee1540-3c6a-4541-b37d-e4ccab3b7ef6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:27.311875 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c983cdca-e02e-43f0-8ed5-9b66a31c5f72 None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:27.312025 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c983cdca-e02e-43f0-8ed5-9b66a31c5f72 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:27.312190 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0ee1540-3c6a-4541-b37d-e4ccab3b7ef6 None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:27.312406 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0ee1540-3c6a-4541-b37d-e4ccab3b7ef6 None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:27.312860 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0ee1540-3c6a-4541-b37d-e4ccab3b7ef6 None admin] PATH_INFO: `/v3/domains` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:27.313518 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0ee1540-3c6a-4541-b37d-e4ccab3b7ef6 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81438) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:54:27.313824 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0ee1540-3c6a-4541-b37d-e4ccab3b7ef6 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:27.315216 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0ee1540-3c6a-4541-b37d-e4ccab3b7ef6 None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:27.332134 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.auth.core [None req-c983cdca-e02e-43f0-8ed5-9b66a31c5f72 None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81437) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:54:27.333323 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-c983cdca-e02e-43f0-8ed5-9b66a31c5f72 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:54:27.341402 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 364/728] 10.222.0.50 () {56 vars in 1109 bytes} [Tue Sep 13 15:54:27 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 188 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:54:27.346991 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0ee1540-3c6a-4541-b37d-e4ccab3b7ef6 None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:27.350041 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:54:27.466290 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 365/729] 10.222.0.50 () {58 vars in 1024 bytes} [Tue Sep 13 15:54:27 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 156 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 13 15:54:27.512460 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3492ca76-7e39-4b7f-bb01-6e2b736f4518 None None] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:27.513108 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3492ca76-7e39-4b7f-bb01-6e2b736f4518 None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:27.513549 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3492ca76-7e39-4b7f-bb01-6e2b736f4518 None None] PATH_INFO: `/` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:27.515125 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 366/730] 10.222.0.50 () {54 vars in 927 bytes} [Tue Sep 13 15:54:27 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 13 15:54:27.515688 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0ee1540-3c6a-4541-b37d-e4ccab3b7ef6 None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:27.521191 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-824b04b2-179c-4909-ad00-354ba6b4a1d1 None None] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:27.521650 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-824b04b2-179c-4909-ad00-354ba6b4a1d1 None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:27.522192 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-824b04b2-179c-4909-ad00-354ba6b4a1d1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:27.555156 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.auth.core [None req-824b04b2-179c-4909-ad00-354ba6b4a1d1 None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81437) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:54:27.558125 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-824b04b2-179c-4909-ad00-354ba6b4a1d1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:54:27.558918 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:54:27.597736 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec08f3f4-e575-44b2-9294-9255f68b3de7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:27.598375 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec08f3f4-e575-44b2-9294-9255f68b3de7 None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:27.598656 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec08f3f4-e575-44b2-9294-9255f68b3de7 None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:27.598948 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec08f3f4-e575-44b2-9294-9255f68b3de7 None admin] PATH_INFO: `/v3/users/swift` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:27.609657 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec08f3f4-e575-44b2-9294-9255f68b3de7 None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:27.611156 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec08f3f4-e575-44b2-9294-9255f68b3de7 None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:27.624334 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application [None req-ec08f3f4-e575-44b2-9294-9255f68b3de7 None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Sep 13 15:54:27.624334 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:54:27.624334 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:54:27.624334 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:54:27.624334 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:54:27.624334 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:54:27.624334 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:54:27.624334 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:54:27.624334 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:54:27.624334 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:54:27.624334 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:54:27.624334 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:54:27.624334 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 13 15:54:27.624334 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 13 15:54:27.624334 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 13 15:54:27.624334 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 13 15:54:27.625307 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:54:27.625307 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:54:27.625307 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 13 15:54:27.625307 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 13 15:54:27.625307 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 13 15:54:27.625307 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 13 15:54:27.625307 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:54:27.625307 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:54:27.625307 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:54:27.625307 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:54:27.625307 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:54:27.625307 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application with Lock( Sep 13 15:54:27.625307 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:54:27.625307 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:54:27.625307 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:54:27.625307 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:54:27.625307 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:54:27.625307 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:54:27.626292 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:54:27.626292 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:54:27.626292 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 13 15:54:27.626292 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 13 15:54:27.626292 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 13 15:54:27.626292 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 13 15:54:27.626292 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 13 15:54:27.626292 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 13 15:54:27.626292 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Sep 13 15:54:27.626292 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Sep 13 15:54:27.626292 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 365/731] 10.222.0.50 () {56 vars in 1096 bytes} [Tue Sep 13 15:54:27 2022] GET /identity/v3/users/swift => generated 83 bytes in 280 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 13 15:54:27.626892 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec08f3f4-e575-44b2-9294-9255f68b3de7 None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:27.667571 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 367/732] 10.222.0.50 () {58 vars in 1024 bytes} [Tue Sep 13 15:54:27 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 147 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 13 15:54:27.668659 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-824b04b2-179c-4909-ad00-354ba6b4a1d1 None None] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:27.794199 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:54:27.798012 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec08f3f4-e575-44b2-9294-9255f68b3de7 None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:27.817075 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-824b04b2-179c-4909-ad00-354ba6b4a1d1 None None] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:27.840535 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6878e235-e153-4597-9304-8f14a7af091d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:27.841383 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6878e235-e153-4597-9304-8f14a7af091d None admin] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:27.842070 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6878e235-e153-4597-9304-8f14a7af091d None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:27.842515 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6878e235-e153-4597-9304-8f14a7af091d None admin] PATH_INFO: `/v3/services` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:27.843668 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6878e235-e153-4597-9304-8f14a7af091d None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:27.845036 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6878e235-e153-4597-9304-8f14a7af091d None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:27.859991 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ffd4e65-3cb4-4b7b-865f-6cb3320fb29e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:27.861025 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ffd4e65-3cb4-4b7b-865f-6cb3320fb29e None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:27.861552 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ffd4e65-3cb4-4b7b-865f-6cb3320fb29e None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:27.862079 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ffd4e65-3cb4-4b7b-865f-6cb3320fb29e None admin] PATH_INFO: `/v3/users` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:27.863126 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ffd4e65-3cb4-4b7b-865f-6cb3320fb29e None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=81437) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:54:27.863650 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ffd4e65-3cb4-4b7b-865f-6cb3320fb29e None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:27.870923 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 366/733] 10.222.0.50 () {60 vars in 1141 bytes} [Tue Sep 13 15:54:27 2022] POST /identity/v3/services => generated 251 bytes in 238 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 15:54:27.870923 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6878e235-e153-4597-9304-8f14a7af091d None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:27.872008 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ffd4e65-3cb4-4b7b-865f-6cb3320fb29e None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:27.902951 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:54:27.904771 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 368/734] 10.222.0.50 () {56 vars in 1135 bytes} [Tue Sep 13 15:54:27 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 236 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:54:27.910826 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ffd4e65-3cb4-4b7b-865f-6cb3320fb29e None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:28.022615 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6878e235-e153-4597-9304-8f14a7af091d None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:28.069776 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ffd4e65-3cb4-4b7b-865f-6cb3320fb29e None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:28.073541 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12daa9df-6309-4220-a353-1572449a7023 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:28.074457 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12daa9df-6309-4220-a353-1572449a7023 None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:28.075012 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12daa9df-6309-4220-a353-1572449a7023 None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:28.075499 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12daa9df-6309-4220-a353-1572449a7023 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:28.090521 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12daa9df-6309-4220-a353-1572449a7023 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 13 15:54:28.096097 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application [None req-12daa9df-6309-4220-a353-1572449a7023 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 13 15:54:28.096097 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:54:28.096097 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:54:28.096097 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:54:28.096097 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:54:28.096097 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:54:28.096097 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:54:28.096097 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:54:28.096097 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:54:28.096097 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:54:28.096097 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:54:28.096097 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:54:28.096097 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 13 15:54:28.096097 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 13 15:54:28.096097 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 13 15:54:28.096097 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 13 15:54:28.097133 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 13 15:54:28.097133 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 13 15:54:28.097133 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 13 15:54:28.097133 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 13 15:54:28.097133 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:54:28.097133 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:54:28.097133 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:54:28.097133 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:54:28.097133 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:54:28.097133 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:54:28.097133 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:54:28.097133 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application with Lock( Sep 13 15:54:28.097133 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:54:28.097133 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:54:28.097133 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:54:28.097133 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:54:28.098001 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:54:28.098001 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:54:28.098001 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:54:28.098001 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:54:28.098001 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 13 15:54:28.098001 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 13 15:54:28.098001 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 13 15:54:28.098001 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 13 15:54:28.098001 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 13 15:54:28.098001 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 13 15:54:28.098001 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 13 15:54:28.098001 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Sep 13 15:54:28.101267 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 367/735] 10.222.0.50 () {56 vars in 1096 bytes} [Tue Sep 13 15:54:27 2022] GET /identity/v3/roles/admin => generated 83 bytes in 235 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:54:28.103374 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12daa9df-6309-4220-a353-1572449a7023 None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:28.112778 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36a17a6d-f23a-493d-b3c1-5fff18f357f9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:28.113556 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36a17a6d-f23a-493d-b3c1-5fff18f357f9 None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:28.113939 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36a17a6d-f23a-493d-b3c1-5fff18f357f9 None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:28.114309 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36a17a6d-f23a-493d-b3c1-5fff18f357f9 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:28.119274 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:54:28.215396 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36a17a6d-f23a-493d-b3c1-5fff18f357f9 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:28.216864 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36a17a6d-f23a-493d-b3c1-5fff18f357f9 None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:28.221367 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:54:28.236709 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12daa9df-6309-4220-a353-1572449a7023 None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:28.278816 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b4061e2-8053-4d77-9159-35ca1a26294f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:28.279938 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b4061e2-8053-4d77-9159-35ca1a26294f None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:28.280429 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b4061e2-8053-4d77-9159-35ca1a26294f None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:28.280908 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b4061e2-8053-4d77-9159-35ca1a26294f None admin] PATH_INFO: `/v3/roles/service` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:28.294526 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b4061e2-8053-4d77-9159-35ca1a26294f None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 13 15:54:28.300017 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application [None req-0b4061e2-8053-4d77-9159-35ca1a26294f None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 13 15:54:28.300017 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:54:28.300017 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:54:28.300017 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:54:28.300017 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:54:28.300017 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:54:28.300017 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:54:28.300017 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:54:28.300017 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:54:28.300017 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:54:28.300017 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:54:28.300017 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:54:28.300017 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 13 15:54:28.300017 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 13 15:54:28.300017 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 13 15:54:28.300017 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 13 15:54:28.301027 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 13 15:54:28.301027 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 13 15:54:28.301027 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 13 15:54:28.301027 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 13 15:54:28.301027 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:54:28.301027 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:54:28.301027 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:54:28.301027 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:54:28.301027 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:54:28.301027 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:54:28.301027 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:54:28.301027 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application with Lock( Sep 13 15:54:28.301027 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:54:28.301027 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:54:28.301027 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:54:28.301027 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:54:28.301960 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:54:28.301960 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:54:28.301960 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:54:28.301960 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:54:28.301960 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 13 15:54:28.301960 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 13 15:54:28.301960 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 13 15:54:28.301960 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 13 15:54:28.301960 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 13 15:54:28.301960 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 13 15:54:28.301960 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 13 15:54:28.301960 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Sep 13 15:54:28.304338 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 368/736] 10.222.0.50 () {56 vars in 1102 bytes} [Tue Sep 13 15:54:28 2022] GET /identity/v3/roles/service => generated 85 bytes in 202 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:54:28.305355 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b4061e2-8053-4d77-9159-35ca1a26294f None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:28.417325 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b4061e2-8053-4d77-9159-35ca1a26294f None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:28.437949 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 369/737] 10.222.0.50 () {58 vars in 1282 bytes} [Tue Sep 13 15:54:27 2022] GET /identity/v3/auth/tokens => generated 903 bytes in 528 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 13 15:54:28.439402 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a668d98-accb-4cd1-898a-7c3915fe53f1 None None] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:28.439683 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a668d98-accb-4cd1-898a-7c3915fe53f1 None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:28.439952 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a668d98-accb-4cd1-898a-7c3915fe53f1 None None] PATH_INFO: `/` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:28.441324 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 370/738] 10.222.0.50 () {54 vars in 927 bytes} [Tue Sep 13 15:54:28 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:54:28.441905 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a668d98-accb-4cd1-898a-7c3915fe53f1 None None] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:28.461093 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff832830-0a66-47f6-9879-ce13e3ac0762 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:28.461730 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff832830-0a66-47f6-9879-ce13e3ac0762 None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:28.462132 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff832830-0a66-47f6-9879-ce13e3ac0762 None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:28.462427 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff832830-0a66-47f6-9879-ce13e3ac0762 None admin] PATH_INFO: `/v3/roles` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:28.463124 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff832830-0a66-47f6-9879-ce13e3ac0762 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81438) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:54:28.463430 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff832830-0a66-47f6-9879-ce13e3ac0762 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:28.464840 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff832830-0a66-47f6-9879-ce13e3ac0762 None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:28.478693 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 369/739] 10.222.0.50 () {56 vars in 1099 bytes} [Tue Sep 13 15:54:28 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:54:28.478693 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff832830-0a66-47f6-9879-ce13e3ac0762 None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:28.486604 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:54:28.490034 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:54:28.559311 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a668d98-accb-4cd1-898a-7c3915fe53f1 None None] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:28.581930 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff832830-0a66-47f6-9879-ce13e3ac0762 None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:28.595518 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ec86108-d2e0-4974-bb38-2175de0efdcc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:28.596124 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ec86108-d2e0-4974-bb38-2175de0efdcc None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:28.596406 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ec86108-d2e0-4974-bb38-2175de0efdcc None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:28.596681 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ec86108-d2e0-4974-bb38-2175de0efdcc None admin] PATH_INFO: `/v3/roles` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:28.597336 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ec86108-d2e0-4974-bb38-2175de0efdcc None admin] RBAC: Adding query filter params (name=service) {{(pid=81437) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:54:28.597645 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ec86108-d2e0-4974-bb38-2175de0efdcc None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:28.599043 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ec86108-d2e0-4974-bb38-2175de0efdcc None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:28.611911 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 371/740] 10.222.0.50 () {56 vars in 1103 bytes} [Tue Sep 13 15:54:28 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:54:28.611911 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb49d418-aaaf-4bf4-a881-df9c0e58e737 None None] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:28.611911 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb49d418-aaaf-4bf4-a881-df9c0e58e737 None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:28.611911 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb49d418-aaaf-4bf4-a881-df9c0e58e737 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:28.622034 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0595ef09-703c-4ff0-ab71-2e2a2fe5d18f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:28.622664 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0595ef09-703c-4ff0-ab71-2e2a2fe5d18f None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:28.622962 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0595ef09-703c-4ff0-ab71-2e2a2fe5d18f None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:28.623235 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0595ef09-703c-4ff0-ab71-2e2a2fe5d18f None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:28.628815 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.auth.core [None req-fb49d418-aaaf-4bf4-a881-df9c0e58e737 None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81437) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:54:28.630009 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-fb49d418-aaaf-4bf4-a881-df9c0e58e737 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:54:28.641104 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0595ef09-703c-4ff0-ab71-2e2a2fe5d18f None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:28.642561 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0595ef09-703c-4ff0-ab71-2e2a2fe5d18f None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:28.658177 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application [None req-0595ef09-703c-4ff0-ab71-2e2a2fe5d18f None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:54:28.658177 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:54:28.658177 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:54:28.658177 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:54:28.658177 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:54:28.658177 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:54:28.658177 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:54:28.658177 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:54:28.658177 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:54:28.658177 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:54:28.658177 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:54:28.658177 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:54:28.658177 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 13 15:54:28.658177 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 13 15:54:28.658177 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 13 15:54:28.658177 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 13 15:54:28.659233 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:54:28.659233 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:54:28.659233 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:54:28.659233 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:54:28.659233 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:54:28.659233 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:54:28.659233 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:54:28.659233 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application with Lock( Sep 13 15:54:28.659233 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:54:28.659233 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:54:28.659233 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:54:28.659233 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:54:28.659233 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:54:28.659233 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:54:28.659233 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:54:28.659233 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:54:28.659233 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 13 15:54:28.660180 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 13 15:54:28.660180 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:54:28.660180 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Sep 13 15:54:28.660180 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 370/741] 10.222.0.50 () {56 vars in 1108 bytes} [Tue Sep 13 15:54:28 2022] GET /identity/v3/domains/Default => generated 87 bytes in 185 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:54:28.660180 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0595ef09-703c-4ff0-ab71-2e2a2fe5d18f None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:28.711454 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 372/742] 10.222.0.50 () {58 vars in 1024 bytes} [Tue Sep 13 15:54:28 2022] POST /identity/v3/auth/tokens => generated 903 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 13 15:54:28.713208 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb49d418-aaaf-4bf4-a881-df9c0e58e737 None None] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:28.729991 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:54:28.753565 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:54:28.775249 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0595ef09-703c-4ff0-ab71-2e2a2fe5d18f None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:28.815004 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8af63ba4-ae3f-4990-b0fd-9e7792f64121 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:28.815004 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8af63ba4-ae3f-4990-b0fd-9e7792f64121 None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:28.815004 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8af63ba4-ae3f-4990-b0fd-9e7792f64121 None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:28.815004 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8af63ba4-ae3f-4990-b0fd-9e7792f64121 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:28.829437 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8af63ba4-ae3f-4990-b0fd-9e7792f64121 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:28.831035 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8af63ba4-ae3f-4990-b0fd-9e7792f64121 None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:28.845447 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb49d418-aaaf-4bf4-a881-df9c0e58e737 None None] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:28.851909 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application [None req-8af63ba4-ae3f-4990-b0fd-9e7792f64121 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:54:28.851909 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:54:28.851909 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:54:28.851909 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:54:28.851909 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:54:28.851909 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:54:28.851909 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:54:28.851909 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:54:28.851909 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:54:28.851909 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:54:28.851909 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:54:28.851909 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:54:28.851909 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 13 15:54:28.851909 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 13 15:54:28.851909 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 13 15:54:28.851909 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 13 15:54:28.853535 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:54:28.853535 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:54:28.853535 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:54:28.853535 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:54:28.853535 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:54:28.853535 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:54:28.853535 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:54:28.853535 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application with Lock( Sep 13 15:54:28.853535 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:54:28.853535 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:54:28.853535 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:54:28.853535 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:54:28.853535 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:54:28.853535 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:54:28.853535 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:54:28.853535 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:54:28.853535 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 13 15:54:28.854422 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 13 15:54:28.854422 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:54:28.854422 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Sep 13 15:54:28.856595 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 371/743] 10.222.0.50 () {56 vars in 1108 bytes} [Tue Sep 13 15:54:28 2022] GET /identity/v3/domains/Default => generated 87 bytes in 197 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:54:28.857586 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8af63ba4-ae3f-4990-b0fd-9e7792f64121 None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:28.888986 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f81e832d-5454-4d2d-8864-89d49aa06090 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:28.889771 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f81e832d-5454-4d2d-8864-89d49aa06090 None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:28.890148 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f81e832d-5454-4d2d-8864-89d49aa06090 None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:28.890522 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f81e832d-5454-4d2d-8864-89d49aa06090 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:28.902454 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f81e832d-5454-4d2d-8864-89d49aa06090 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:28.904427 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f81e832d-5454-4d2d-8864-89d49aa06090 None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:28.920429 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application [None req-f81e832d-5454-4d2d-8864-89d49aa06090 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:54:28.920429 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:54:28.920429 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:54:28.920429 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:54:28.920429 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:54:28.920429 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:54:28.920429 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:54:28.920429 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:54:28.920429 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:54:28.920429 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:54:28.920429 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:54:28.920429 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:54:28.920429 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 13 15:54:28.920429 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 13 15:54:28.920429 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 13 15:54:28.920429 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 13 15:54:28.921593 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:54:28.921593 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:54:28.921593 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:54:28.921593 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:54:28.921593 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:54:28.921593 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:54:28.921593 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:54:28.921593 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application with Lock( Sep 13 15:54:28.921593 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:54:28.921593 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:54:28.921593 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:54:28.921593 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:54:28.921593 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:54:28.921593 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:54:28.921593 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:54:28.921593 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:54:28.921593 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 13 15:54:28.922679 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 13 15:54:28.922679 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:54:28.922679 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application Sep 13 15:54:28.923993 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 373/744] 10.222.0.50 () {56 vars in 1108 bytes} [Tue Sep 13 15:54:28 2022] GET /identity/v3/domains/Default => generated 87 bytes in 212 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:54:28.927802 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f81e832d-5454-4d2d-8864-89d49aa06090 None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:28.982810 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8af63ba4-ae3f-4990-b0fd-9e7792f64121 None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:28.997907 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:54:29.022030 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9aae514-f1ec-4319-b6b4-4635c29420b9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:29.022641 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9aae514-f1ec-4319-b6b4-4635c29420b9 None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:29.022953 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9aae514-f1ec-4319-b6b4-4635c29420b9 None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:29.023239 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9aae514-f1ec-4319-b6b4-4635c29420b9 None admin] PATH_INFO: `/v3/domains` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:29.023882 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9aae514-f1ec-4319-b6b4-4635c29420b9 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81438) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:54:29.024725 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9aae514-f1ec-4319-b6b4-4635c29420b9 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:29.026100 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9aae514-f1ec-4319-b6b4-4635c29420b9 None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:29.028639 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:54:29.042793 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 372/745] 10.222.0.50 () {56 vars in 1109 bytes} [Tue Sep 13 15:54:28 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:54:29.042793 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9aae514-f1ec-4319-b6b4-4635c29420b9 None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:29.056440 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f81e832d-5454-4d2d-8864-89d49aa06090 None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:29.095190 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a05dee13-b202-40e3-b589-4f0424e94fda None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:29.095792 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a05dee13-b202-40e3-b589-4f0424e94fda None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:29.096074 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a05dee13-b202-40e3-b589-4f0424e94fda None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:29.096348 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a05dee13-b202-40e3-b589-4f0424e94fda None admin] PATH_INFO: `/v3/domains` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:29.096985 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a05dee13-b202-40e3-b589-4f0424e94fda None admin] RBAC: Adding query filter params (name=Default) {{(pid=81437) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:54:29.097288 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a05dee13-b202-40e3-b589-4f0424e94fda None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:29.098822 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a05dee13-b202-40e3-b589-4f0424e94fda None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:29.111166 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 374/746] 10.222.0.50 () {56 vars in 1109 bytes} [Tue Sep 13 15:54:28 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 187 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:54:29.116091 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a05dee13-b202-40e3-b589-4f0424e94fda None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:29.148438 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9aae514-f1ec-4319-b6b4-4635c29420b9 None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:29.184778 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00556168-63ee-4380-b52a-db9dbae38062 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:29.185389 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00556168-63ee-4380-b52a-db9dbae38062 None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:29.185669 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00556168-63ee-4380-b52a-db9dbae38062 None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:29.185943 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00556168-63ee-4380-b52a-db9dbae38062 None admin] PATH_INFO: `/v3/domains` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:29.186581 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00556168-63ee-4380-b52a-db9dbae38062 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81438) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:54:29.186886 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00556168-63ee-4380-b52a-db9dbae38062 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:29.188278 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00556168-63ee-4380-b52a-db9dbae38062 None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:29.201097 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 373/747] 10.222.0.50 () {56 vars in 1109 bytes} [Tue Sep 13 15:54:29 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:54:29.201830 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00556168-63ee-4380-b52a-db9dbae38062 None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:29.225579 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a05dee13-b202-40e3-b589-4f0424e94fda None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:29.244588 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:54:29.262638 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03559409-9013-498b-bdcd-ba17dcea1052 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:29.263277 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03559409-9013-498b-bdcd-ba17dcea1052 None admin] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:29.263556 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03559409-9013-498b-bdcd-ba17dcea1052 None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:29.263827 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03559409-9013-498b-bdcd-ba17dcea1052 None admin] PATH_INFO: `/v3/services` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:29.264468 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03559409-9013-498b-bdcd-ba17dcea1052 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:29.265817 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03559409-9013-498b-bdcd-ba17dcea1052 None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:29.277002 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:54:29.284122 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 375/748] 10.222.0.50 () {60 vars in 1141 bytes} [Tue Sep 13 15:54:29 2022] POST /identity/v3/services => generated 240 bytes in 172 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 15:54:29.285142 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03559409-9013-498b-bdcd-ba17dcea1052 None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:29.327260 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00556168-63ee-4380-b52a-db9dbae38062 None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:29.376514 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdc5cbfc-f79f-46f6-a856-29e0ed878fae None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:29.377449 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdc5cbfc-f79f-46f6-a856-29e0ed878fae None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:29.377871 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdc5cbfc-f79f-46f6-a856-29e0ed878fae None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:29.378317 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdc5cbfc-f79f-46f6-a856-29e0ed878fae None admin] PATH_INFO: `/v3/projects/service` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:29.388676 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdc5cbfc-f79f-46f6-a856-29e0ed878fae None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:29.390374 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdc5cbfc-f79f-46f6-a856-29e0ed878fae None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:29.405992 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application [None req-bdc5cbfc-f79f-46f6-a856-29e0ed878fae None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 13 15:54:29.405992 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:54:29.405992 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:54:29.405992 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:54:29.405992 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:54:29.405992 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:54:29.405992 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:54:29.405992 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:54:29.405992 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:54:29.405992 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:54:29.405992 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:54:29.405992 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:54:29.405992 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 13 15:54:29.405992 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 13 15:54:29.405992 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 13 15:54:29.407136 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 13 15:54:29.407136 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:54:29.407136 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:54:29.407136 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:54:29.407136 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:54:29.407136 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:54:29.407136 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:54:29.407136 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:54:29.407136 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application with Lock( Sep 13 15:54:29.407136 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:54:29.407136 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:54:29.407136 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:54:29.407136 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:54:29.407136 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:54:29.407136 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:54:29.407136 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:54:29.407136 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:54:29.408262 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 13 15:54:29.408262 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 13 15:54:29.408262 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 13 15:54:29.408262 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 13 15:54:29.408262 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 13 15:54:29.408262 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 13 15:54:29.408262 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 13 15:54:29.408262 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Sep 13 15:54:29.410364 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 374/749] 10.222.0.50 () {56 vars in 1111 bytes} [Tue Sep 13 15:54:29 2022] GET /identity/v3/projects/service => generated 88 bytes in 209 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:54:29.411425 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdc5cbfc-f79f-46f6-a856-29e0ed878fae None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:29.418092 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03559409-9013-498b-bdcd-ba17dcea1052 None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:29.461629 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cb9f335-0327-4b39-af04-2c6d36087aad None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:29.462946 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cb9f335-0327-4b39-af04-2c6d36087aad None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:29.463289 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cb9f335-0327-4b39-af04-2c6d36087aad None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:29.463566 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cb9f335-0327-4b39-af04-2c6d36087aad None admin] PATH_INFO: `/v3/users/nova` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:29.474593 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5cb9f335-0327-4b39-af04-2c6d36087aad None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:29.476057 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5cb9f335-0327-4b39-af04-2c6d36087aad None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:29.487303 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application [None req-5cb9f335-0327-4b39-af04-2c6d36087aad None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Sep 13 15:54:29.487303 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:54:29.487303 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:54:29.487303 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:54:29.487303 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:54:29.487303 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:54:29.487303 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:54:29.487303 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:54:29.487303 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:54:29.487303 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:54:29.487303 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:54:29.487303 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:54:29.487303 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 13 15:54:29.487303 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 13 15:54:29.487303 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 13 15:54:29.487303 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 13 15:54:29.488275 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:54:29.488275 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:54:29.488275 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 13 15:54:29.488275 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 13 15:54:29.488275 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 13 15:54:29.488275 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 13 15:54:29.488275 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:54:29.488275 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:54:29.488275 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:54:29.488275 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:54:29.488275 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:54:29.488275 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application with Lock( Sep 13 15:54:29.488275 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:54:29.488275 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:54:29.488275 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:54:29.488275 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:54:29.488275 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:54:29.488275 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:54:29.489826 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:54:29.489826 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:54:29.489826 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 13 15:54:29.489826 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 13 15:54:29.489826 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 13 15:54:29.489826 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 13 15:54:29.489826 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 13 15:54:29.489826 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 13 15:54:29.489826 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Sep 13 15:54:29.489826 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application Sep 13 15:54:29.489826 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 376/750] 10.222.0.50 () {56 vars in 1093 bytes} [Tue Sep 13 15:54:29 2022] GET /identity/v3/users/nova => generated 82 bytes in 204 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:54:29.489826 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cb9f335-0327-4b39-af04-2c6d36087aad None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:29.543713 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:54:29.556931 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdc5cbfc-f79f-46f6-a856-29e0ed878fae None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:29.562700 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:54:29.601204 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac3a2ca0-e008-435e-b007-11966530189a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:29.602464 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac3a2ca0-e008-435e-b007-11966530189a None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:29.602917 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac3a2ca0-e008-435e-b007-11966530189a None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:29.603365 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac3a2ca0-e008-435e-b007-11966530189a None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:29.605634 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cb9f335-0327-4b39-af04-2c6d36087aad None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:29.613313 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac3a2ca0-e008-435e-b007-11966530189a None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:29.615344 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac3a2ca0-e008-435e-b007-11966530189a None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:29.629757 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application [None req-ac3a2ca0-e008-435e-b007-11966530189a None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Sep 13 15:54:29.629757 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:54:29.629757 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:54:29.629757 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:54:29.629757 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:54:29.629757 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:54:29.629757 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:54:29.629757 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:54:29.629757 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:54:29.629757 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:54:29.629757 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:54:29.629757 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:54:29.629757 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 13 15:54:29.629757 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 13 15:54:29.629757 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 13 15:54:29.629757 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 13 15:54:29.630785 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:54:29.630785 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:54:29.630785 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 13 15:54:29.630785 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 13 15:54:29.630785 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 13 15:54:29.630785 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 13 15:54:29.630785 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:54:29.630785 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:54:29.630785 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:54:29.630785 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:54:29.630785 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:54:29.630785 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application with Lock( Sep 13 15:54:29.630785 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:54:29.630785 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:54:29.630785 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:54:29.630785 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:54:29.630785 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:54:29.630785 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:54:29.631780 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:54:29.631780 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:54:29.631780 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 13 15:54:29.631780 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 13 15:54:29.631780 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 13 15:54:29.631780 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 13 15:54:29.631780 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 13 15:54:29.631780 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 13 15:54:29.631780 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Sep 13 15:54:29.631780 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Sep 13 15:54:29.632965 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 375/751] 10.222.0.50 () {56 vars in 1117 bytes} [Tue Sep 13 15:54:29 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 221 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:54:29.633491 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac3a2ca0-e008-435e-b007-11966530189a None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:29.647554 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-919042e8-fb4e-432f-9f72-5bb4229051ae None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:29.648865 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-919042e8-fb4e-432f-9f72-5bb4229051ae None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:29.649922 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-919042e8-fb4e-432f-9f72-5bb4229051ae None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:29.650403 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-919042e8-fb4e-432f-9f72-5bb4229051ae None admin] PATH_INFO: `/v3/projects` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:29.652073 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-919042e8-fb4e-432f-9f72-5bb4229051ae None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81437) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:54:29.652623 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-919042e8-fb4e-432f-9f72-5bb4229051ae None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:29.654792 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-919042e8-fb4e-432f-9f72-5bb4229051ae None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:29.674635 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 377/752] 10.222.0.50 () {56 vars in 1148 bytes} [Tue Sep 13 15:54:29 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:54:29.675972 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-919042e8-fb4e-432f-9f72-5bb4229051ae None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:29.752253 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac3a2ca0-e008-435e-b007-11966530189a None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:29.792408 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd84d38e-84a2-4ba2-b8bf-00779973477d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:29.793023 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd84d38e-84a2-4ba2-b8bf-00779973477d None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:29.793299 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd84d38e-84a2-4ba2-b8bf-00779973477d None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:29.793572 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd84d38e-84a2-4ba2-b8bf-00779973477d None admin] PATH_INFO: `/v3/users` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:29.794348 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd84d38e-84a2-4ba2-b8bf-00779973477d None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81438) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:54:29.794592 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd84d38e-84a2-4ba2-b8bf-00779973477d None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:29.796154 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd84d38e-84a2-4ba2-b8bf-00779973477d None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:29.816164 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-919042e8-fb4e-432f-9f72-5bb4229051ae None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:29.816833 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 376/753] 10.222.0.50 () {56 vars in 1133 bytes} [Tue Sep 13 15:54:29 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:54:29.817472 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd84d38e-84a2-4ba2-b8bf-00779973477d None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:29.828927 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:54:29.844188 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:54:29.864610 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-555fa4ed-58be-4777-b279-d0981c6dc8bc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:29.865274 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-555fa4ed-58be-4777-b279-d0981c6dc8bc None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:29.865578 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-555fa4ed-58be-4777-b279-d0981c6dc8bc None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:29.865859 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-555fa4ed-58be-4777-b279-d0981c6dc8bc None admin] PATH_INFO: `/v3/users` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:29.866602 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-555fa4ed-58be-4777-b279-d0981c6dc8bc None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=81437) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:54:29.866924 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-555fa4ed-58be-4777-b279-d0981c6dc8bc None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:29.869996 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-555fa4ed-58be-4777-b279-d0981c6dc8bc None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:29.893663 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 378/754] 10.222.0.50 () {56 vars in 1149 bytes} [Tue Sep 13 15:54:29 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 218 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:54:29.894864 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-555fa4ed-58be-4777-b279-d0981c6dc8bc None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:29.948207 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd84d38e-84a2-4ba2-b8bf-00779973477d None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:30.002609 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b1365ce-c9a4-49a2-877f-2bf3d426f1a2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:30.003391 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b1365ce-c9a4-49a2-877f-2bf3d426f1a2 None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:30.003673 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b1365ce-c9a4-49a2-877f-2bf3d426f1a2 None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:30.003948 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b1365ce-c9a4-49a2-877f-2bf3d426f1a2 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:30.004930 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b1365ce-c9a4-49a2-877f-2bf3d426f1a2 None admin] RBAC: Adding query filter params (role.id=031d4e1a23e34f71858fc4a6dfbd4ccb, scope.project.id=5811e3b83a1e462e9320cad79a3ad791, user.id=b00ab0d6321d483188bc69deaddcdbbc) {{(pid=81438) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:54:30.005251 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b1365ce-c9a4-49a2-877f-2bf3d426f1a2 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:30.007434 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b1365ce-c9a4-49a2-877f-2bf3d426f1a2 None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:30.015371 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-555fa4ed-58be-4777-b279-d0981c6dc8bc None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:30.015528 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 377/755] 10.222.0.50 () {56 vars in 1374 bytes} [Tue Sep 13 15:54:29 2022] GET /identity/v3/role_assignments?user.id=b00ab0d6321d483188bc69deaddcdbbc&scope.project.id=5811e3b83a1e462e9320cad79a3ad791&role.id=031d4e1a23e34f71858fc4a6dfbd4ccb => generated 259 bytes in 198 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:54:30.016373 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b1365ce-c9a4-49a2-877f-2bf3d426f1a2 None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:30.056798 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61fbd602-ced9-42bc-b9c8-7628c85cc993 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:30.058104 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61fbd602-ced9-42bc-b9c8-7628c85cc993 None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:30.058597 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61fbd602-ced9-42bc-b9c8-7628c85cc993 None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:30.059112 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61fbd602-ced9-42bc-b9c8-7628c85cc993 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:30.072004 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61fbd602-ced9-42bc-b9c8-7628c85cc993 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:30.073527 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61fbd602-ced9-42bc-b9c8-7628c85cc993 None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:30.090881 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application [None req-61fbd602-ced9-42bc-b9c8-7628c85cc993 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:54:30.090881 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:54:30.090881 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:54:30.090881 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:54:30.090881 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:54:30.090881 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:54:30.090881 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:54:30.090881 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:54:30.090881 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:54:30.090881 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:54:30.090881 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:54:30.090881 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:54:30.090881 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 13 15:54:30.090881 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 13 15:54:30.090881 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 13 15:54:30.090881 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 13 15:54:30.092049 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:54:30.092049 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:54:30.092049 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:54:30.092049 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:54:30.092049 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:54:30.092049 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:54:30.092049 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:54:30.092049 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application with Lock( Sep 13 15:54:30.092049 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:54:30.092049 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:54:30.092049 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:54:30.092049 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:54:30.092049 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:54:30.092049 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:54:30.092049 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:54:30.092049 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:54:30.092049 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 13 15:54:30.093590 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 13 15:54:30.093590 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:54:30.093590 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application Sep 13 15:54:30.093590 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 379/756] 10.222.0.50 () {56 vars in 1108 bytes} [Tue Sep 13 15:54:29 2022] GET /identity/v3/domains/Default => generated 87 bytes in 198 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:54:30.097969 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61fbd602-ced9-42bc-b9c8-7628c85cc993 None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:30.151286 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:54:30.186281 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b1365ce-c9a4-49a2-877f-2bf3d426f1a2 None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:30.189754 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:54:30.230575 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-575d3839-095a-4726-907c-f445b7877f5b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:30.231239 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-575d3839-095a-4726-907c-f445b7877f5b None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:30.231523 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-575d3839-095a-4726-907c-f445b7877f5b None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:30.231798 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-575d3839-095a-4726-907c-f445b7877f5b None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:30.309559 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61fbd602-ced9-42bc-b9c8-7628c85cc993 None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:30.325878 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-575d3839-095a-4726-907c-f445b7877f5b None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:30.327388 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-575d3839-095a-4726-907c-f445b7877f5b None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:30.359553 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79992a99-36df-4b79-b33a-420c5cf708de None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:30.360692 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79992a99-36df-4b79-b33a-420c5cf708de None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:30.361153 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79992a99-36df-4b79-b33a-420c5cf708de None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:30.361603 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79992a99-36df-4b79-b33a-420c5cf708de None admin] PATH_INFO: `/v3/domains` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:30.362599 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79992a99-36df-4b79-b33a-420c5cf708de None admin] RBAC: Adding query filter params (name=Default) {{(pid=81437) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:54:30.363104 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79992a99-36df-4b79-b33a-420c5cf708de None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:30.365412 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79992a99-36df-4b79-b33a-420c5cf708de None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:30.380777 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 380/757] 10.222.0.50 () {56 vars in 1109 bytes} [Tue Sep 13 15:54:30 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 283 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 13 15:54:30.382370 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85716b9c-f212-4284-89d4-3c8df772e7ce None None] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:30.382821 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85716b9c-f212-4284-89d4-3c8df772e7ce None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:30.383308 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85716b9c-f212-4284-89d4-3c8df772e7ce None None] PATH_INFO: `/` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:30.389454 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 381/758] 10.222.0.50 () {54 vars in 927 bytes} [Tue Sep 13 15:54:30 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:54:30.390607 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85716b9c-f212-4284-89d4-3c8df772e7ce None None] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:30.408598 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:54:30.456456 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 378/759] 10.222.0.50 () {58 vars in 1282 bytes} [Tue Sep 13 15:54:30 2022] GET /identity/v3/auth/tokens => generated 1002 bytes in 436 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 13 15:54:30.456456 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e11ae7a8-08fa-4108-b8e1-4a0866c8a17a None None] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:30.456755 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e11ae7a8-08fa-4108-b8e1-4a0866c8a17a None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:30.457041 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e11ae7a8-08fa-4108-b8e1-4a0866c8a17a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:30.476122 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.auth.core [None req-e11ae7a8-08fa-4108-b8e1-4a0866c8a17a None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81438) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:54:30.478153 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-e11ae7a8-08fa-4108-b8e1-4a0866c8a17a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:54:30.514053 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:54:30.530937 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85716b9c-f212-4284-89d4-3c8df772e7ce None None] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:30.554981 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 379/760] 10.222.0.50 () {58 vars in 1024 bytes} [Tue Sep 13 15:54:30 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 13 15:54:30.556206 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e11ae7a8-08fa-4108-b8e1-4a0866c8a17a None None] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:30.593351 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1141e899-c505-4f2e-9d14-39e1e8559372 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:30.594573 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1141e899-c505-4f2e-9d14-39e1e8559372 None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:30.595104 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1141e899-c505-4f2e-9d14-39e1e8559372 None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:30.595957 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1141e899-c505-4f2e-9d14-39e1e8559372 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:30.603876 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1141e899-c505-4f2e-9d14-39e1e8559372 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:30.605565 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1141e899-c505-4f2e-9d14-39e1e8559372 None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:30.618394 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application [None req-1141e899-c505-4f2e-9d14-39e1e8559372 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 13 15:54:30.618394 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:54:30.618394 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:54:30.618394 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:54:30.618394 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:54:30.618394 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:54:30.618394 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:54:30.618394 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:54:30.618394 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:54:30.618394 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:54:30.618394 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:54:30.618394 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:54:30.618394 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 13 15:54:30.618394 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 13 15:54:30.618394 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 13 15:54:30.619325 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 13 15:54:30.619325 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:54:30.619325 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:54:30.619325 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:54:30.619325 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:54:30.619325 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:54:30.619325 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:54:30.619325 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:54:30.619325 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application with Lock( Sep 13 15:54:30.619325 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:54:30.619325 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:54:30.619325 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:54:30.619325 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:54:30.619325 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:54:30.619325 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:54:30.619325 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:54:30.619325 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:54:30.620215 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 13 15:54:30.620215 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 13 15:54:30.620215 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 13 15:54:30.620215 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 13 15:54:30.620215 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 13 15:54:30.620215 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 13 15:54:30.620215 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 13 15:54:30.620215 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application Sep 13 15:54:30.622466 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 382/761] 10.222.0.50 () {56 vars in 1111 bytes} [Tue Sep 13 15:54:30 2022] GET /identity/v3/projects/service => generated 88 bytes in 232 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:54:30.627676 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1141e899-c505-4f2e-9d14-39e1e8559372 None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:30.725694 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e11ae7a8-08fa-4108-b8e1-4a0866c8a17a None None] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:30.728064 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:54:30.767886 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbc9be8c-bfa4-4b95-8ba4-8701e9cc4777 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:30.768885 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbc9be8c-bfa4-4b95-8ba4-8701e9cc4777 None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:30.769390 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbc9be8c-bfa4-4b95-8ba4-8701e9cc4777 None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:30.769857 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbc9be8c-bfa4-4b95-8ba4-8701e9cc4777 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:30.783104 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbc9be8c-bfa4-4b95-8ba4-8701e9cc4777 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:30.784841 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbc9be8c-bfa4-4b95-8ba4-8701e9cc4777 None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:30.801533 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application [None req-bbc9be8c-bfa4-4b95-8ba4-8701e9cc4777 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:54:30.801533 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:54:30.801533 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:54:30.801533 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:54:30.801533 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:54:30.801533 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:54:30.801533 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:54:30.801533 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:54:30.801533 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:54:30.801533 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:54:30.801533 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:54:30.801533 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:54:30.801533 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 13 15:54:30.801533 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 13 15:54:30.801533 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 13 15:54:30.801533 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 13 15:54:30.802529 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:54:30.802529 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:54:30.802529 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:54:30.802529 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:54:30.802529 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:54:30.802529 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:54:30.802529 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:54:30.802529 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application with Lock( Sep 13 15:54:30.802529 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:54:30.802529 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:54:30.802529 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:54:30.802529 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:54:30.802529 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:54:30.802529 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:54:30.802529 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:54:30.802529 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:54:30.802529 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 13 15:54:30.803489 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 13 15:54:30.803489 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:54:30.803489 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Sep 13 15:54:30.805603 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 380/762] 10.222.0.50 () {56 vars in 1108 bytes} [Tue Sep 13 15:54:30 2022] GET /identity/v3/domains/Default => generated 87 bytes in 250 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:54:30.811511 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbc9be8c-bfa4-4b95-8ba4-8701e9cc4777 None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:30.814279 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1141e899-c505-4f2e-9d14-39e1e8559372 None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:30.888802 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e7b139b-efdc-4974-9162-ec00bd5c85bd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:30.889425 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e7b139b-efdc-4974-9162-ec00bd5c85bd None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:30.889677 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e7b139b-efdc-4974-9162-ec00bd5c85bd None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:30.889869 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e7b139b-efdc-4974-9162-ec00bd5c85bd None admin] PATH_INFO: `/v3/projects` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:30.890564 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e7b139b-efdc-4974-9162-ec00bd5c85bd None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81437) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:54:30.890872 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e7b139b-efdc-4974-9162-ec00bd5c85bd None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:30.892400 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e7b139b-efdc-4974-9162-ec00bd5c85bd None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:30.921172 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 383/763] 10.222.0.50 () {56 vars in 1148 bytes} [Tue Sep 13 15:54:30 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 294 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 13 15:54:30.923004 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbc9be8c-bfa4-4b95-8ba4-8701e9cc4777 None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:30.926807 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e7b139b-efdc-4974-9162-ec00bd5c85bd None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:30.947155 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:54:30.963060 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed2d2747-c4e9-4ab4-a357-db3bb2e599cb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:30.964023 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed2d2747-c4e9-4ab4-a357-db3bb2e599cb None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:30.964518 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed2d2747-c4e9-4ab4-a357-db3bb2e599cb None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:30.964981 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed2d2747-c4e9-4ab4-a357-db3bb2e599cb None admin] PATH_INFO: `/v3/domains` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:30.965926 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed2d2747-c4e9-4ab4-a357-db3bb2e599cb None admin] RBAC: Adding query filter params (name=Default) {{(pid=81438) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:54:30.966422 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed2d2747-c4e9-4ab4-a357-db3bb2e599cb None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:30.968030 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed2d2747-c4e9-4ab4-a357-db3bb2e599cb None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:30.985124 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 381/764] 10.222.0.50 () {56 vars in 1109 bytes} [Tue Sep 13 15:54:30 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 13 15:54:30.990567 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed2d2747-c4e9-4ab4-a357-db3bb2e599cb None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:31.009645 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:54:31.067955 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e7b139b-efdc-4974-9162-ec00bd5c85bd None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:31.111887 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c8608b0-b6af-4cf7-882c-5049a61db764 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:31.112831 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c8608b0-b6af-4cf7-882c-5049a61db764 None admin] REQUEST_METHOD: `PUT` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:31.113185 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c8608b0-b6af-4cf7-882c-5049a61db764 None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:31.113581 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c8608b0-b6af-4cf7-882c-5049a61db764 None admin] PATH_INFO: `/v3/projects/5811e3b83a1e462e9320cad79a3ad791/users/4e039b46c5e3445bbb39904b00c5430e/roles/031d4e1a23e34f71858fc4a6dfbd4ccb` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:31.114722 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed2d2747-c4e9-4ab4-a357-db3bb2e599cb None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:31.115633 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c8608b0-b6af-4cf7-882c-5049a61db764 None admin] RBAC: Authorizing `identity:create_grant(project_id=5811e3b83a1e462e9320cad79a3ad791, user_id=4e039b46c5e3445bbb39904b00c5430e, role_id=031d4e1a23e34f71858fc4a6dfbd4ccb)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:31.118858 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c8608b0-b6af-4cf7-882c-5049a61db764 None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:31.143225 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 384/765] 10.222.0.50 () {58 vars in 1439 bytes} [Tue Sep 13 15:54:30 2022] PUT /identity/v3/projects/5811e3b83a1e462e9320cad79a3ad791/users/4e039b46c5e3445bbb39904b00c5430e/roles/031d4e1a23e34f71858fc4a6dfbd4ccb => generated 0 bytes in 207 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 13 15:54:31.144395 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c8608b0-b6af-4cf7-882c-5049a61db764 None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:31.155691 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-241f4419-62a4-4f1e-bf65-a7105a535df8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:31.156729 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-241f4419-62a4-4f1e-bf65-a7105a535df8 None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:31.157208 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-241f4419-62a4-4f1e-bf65-a7105a535df8 None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:31.157675 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-241f4419-62a4-4f1e-bf65-a7105a535df8 None admin] PATH_INFO: `/v3/services/block-storage` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:31.158626 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-241f4419-62a4-4f1e-bf65-a7105a535df8 None admin] RBAC: Authorizing `identity:get_service(service_id=block-storage)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:31.160227 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-241f4419-62a4-4f1e-bf65-a7105a535df8 None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:31.170421 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application [None req-241f4419-62a4-4f1e-bf65-a7105a535df8 None admin] Could not find service: block-storage.: keystone.exception.ServiceNotFound: Could not find service: block-storage. Sep 13 15:54:31.170421 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:54:31.170421 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 13 15:54:31.170421 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 13 15:54:31.170421 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 13 15:54:31.170421 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 13 15:54:31.170421 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 13 15:54:31.170421 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 13 15:54:31.170421 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Sep 13 15:54:31.170421 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Sep 13 15:54:31.170421 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 13 15:54:31.170421 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Sep 13 15:54:31.170421 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:54:31.170421 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:54:31.170421 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:54:31.170421 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:54:31.170421 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:54:31.171570 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:54:31.171570 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:54:31.171570 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:54:31.171570 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:54:31.171570 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:54:31.171570 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:54:31.171570 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 13 15:54:31.171570 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 13 15:54:31.171570 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 13 15:54:31.171570 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 13 15:54:31.171570 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:54:31.171570 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:54:31.171570 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:54:31.171570 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:54:31.171570 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:54:31.171570 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:54:31.172402 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:54:31.172402 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application with Lock( Sep 13 15:54:31.172402 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:54:31.172402 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:54:31.172402 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:54:31.172402 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:54:31.172402 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:54:31.172402 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:54:31.172402 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:54:31.172402 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:54:31.172402 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 13 15:54:31.172402 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 13 15:54:31.172402 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Sep 13 15:54:31.172402 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Sep 13 15:54:31.175509 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 382/766] 10.222.0.50 () {56 vars in 1129 bytes} [Tue Sep 13 15:54:30 2022] GET /identity/v3/services/block-storage => generated 94 bytes in 189 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:54:31.203625 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-241f4419-62a4-4f1e-bf65-a7105a535df8 None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:31.244664 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:54:31.282179 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c8608b0-b6af-4cf7-882c-5049a61db764 None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:31.325189 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fcee1d5-7133-4568-95d0-e4867989caf2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:31.325189 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fcee1d5-7133-4568-95d0-e4867989caf2 None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:31.325189 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fcee1d5-7133-4568-95d0-e4867989caf2 None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:31.325189 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fcee1d5-7133-4568-95d0-e4867989caf2 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:31.331343 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fcee1d5-7133-4568-95d0-e4867989caf2 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:31.332903 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fcee1d5-7133-4568-95d0-e4867989caf2 None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:31.345515 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application [None req-5fcee1d5-7133-4568-95d0-e4867989caf2 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 13 15:54:31.345515 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:54:31.345515 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:54:31.345515 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:54:31.345515 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:54:31.345515 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:54:31.345515 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:54:31.345515 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:54:31.345515 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:54:31.345515 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:54:31.345515 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:54:31.345515 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:54:31.345515 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 13 15:54:31.345515 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 13 15:54:31.345515 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 13 15:54:31.346473 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 13 15:54:31.346473 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:54:31.346473 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:54:31.346473 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:54:31.346473 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:54:31.346473 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:54:31.346473 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:54:31.346473 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:54:31.346473 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application with Lock( Sep 13 15:54:31.346473 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:54:31.346473 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:54:31.346473 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:54:31.346473 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:54:31.346473 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:54:31.346473 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:54:31.346473 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:54:31.346473 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:54:31.347412 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 13 15:54:31.347412 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 13 15:54:31.347412 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 13 15:54:31.347412 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 13 15:54:31.347412 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 13 15:54:31.347412 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 13 15:54:31.347412 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 13 15:54:31.347412 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application Sep 13 15:54:31.351108 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 385/767] 10.222.0.50 () {56 vars in 1111 bytes} [Tue Sep 13 15:54:31 2022] GET /identity/v3/projects/service => generated 88 bytes in 203 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:54:31.353247 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fcee1d5-7133-4568-95d0-e4867989caf2 None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:31.363385 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:54:31.402985 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-241f4419-62a4-4f1e-bf65-a7105a535df8 None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:31.469162 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fcee1d5-7133-4568-95d0-e4867989caf2 None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:31.495062 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b58485fe-1764-4587-8204-3a143b6035dc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:31.496162 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b58485fe-1764-4587-8204-3a143b6035dc None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:31.496702 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b58485fe-1764-4587-8204-3a143b6035dc None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:31.497171 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b58485fe-1764-4587-8204-3a143b6035dc None admin] PATH_INFO: `/v3/services` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:31.497990 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b58485fe-1764-4587-8204-3a143b6035dc None admin] RBAC: Adding query filter params (name=block-storage) {{(pid=81438) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:54:31.498598 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b58485fe-1764-4587-8204-3a143b6035dc None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:31.500381 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b58485fe-1764-4587-8204-3a143b6035dc None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:31.501753 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:54:31.507109 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 383/768] 10.222.0.50 () {56 vars in 1124 bytes} [Tue Sep 13 15:54:31 2022] GET /identity/v3/services?name=block-storage => generated 130 bytes in 304 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:54:31.513686 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-093e3395-cca7-442d-9b84-0e39c8a35896 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:31.514842 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-093e3395-cca7-442d-9b84-0e39c8a35896 None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:31.515214 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-093e3395-cca7-442d-9b84-0e39c8a35896 None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:31.515500 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-093e3395-cca7-442d-9b84-0e39c8a35896 None admin] PATH_INFO: `/v3/projects` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:31.516204 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-093e3395-cca7-442d-9b84-0e39c8a35896 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81437) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:54:31.516512 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-093e3395-cca7-442d-9b84-0e39c8a35896 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:31.517974 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-093e3395-cca7-442d-9b84-0e39c8a35896 None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:31.524645 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b58485fe-1764-4587-8204-3a143b6035dc None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:31.532829 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 386/769] 10.222.0.50 () {56 vars in 1148 bytes} [Tue Sep 13 15:54:31 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 180 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 13 15:54:31.538938 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-093e3395-cca7-442d-9b84-0e39c8a35896 None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:31.637037 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b58485fe-1764-4587-8204-3a143b6035dc None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:31.655351 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-093e3395-cca7-442d-9b84-0e39c8a35896 None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:31.675863 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d7981ba-2b06-4c7c-9465-c0121e11ebdf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:31.676824 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d7981ba-2b06-4c7c-9465-c0121e11ebdf None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:31.677333 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d7981ba-2b06-4c7c-9465-c0121e11ebdf None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:31.677849 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d7981ba-2b06-4c7c-9465-c0121e11ebdf None admin] PATH_INFO: `/v3/services` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:31.678689 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d7981ba-2b06-4c7c-9465-c0121e11ebdf None admin] RBAC: Adding query filter params (type=block-storage) {{(pid=81438) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:54:31.679235 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d7981ba-2b06-4c7c-9465-c0121e11ebdf None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:31.680931 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d7981ba-2b06-4c7c-9465-c0121e11ebdf None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:31.688196 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 384/770] 10.222.0.50 () {56 vars in 1124 bytes} [Tue Sep 13 15:54:31 2022] GET /identity/v3/services?type=block-storage => generated 367 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 13 15:54:31.692675 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52deae3e-5d5d-43a3-b349-207b060a05c2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:31.693396 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52deae3e-5d5d-43a3-b349-207b060a05c2 None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:31.693504 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52deae3e-5d5d-43a3-b349-207b060a05c2 None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:31.693787 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52deae3e-5d5d-43a3-b349-207b060a05c2 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:31.694512 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52deae3e-5d5d-43a3-b349-207b060a05c2 None admin] RBAC: Adding query filter params (role.id=da847124d9374ac88b27968451c486ab, scope.project.id=5811e3b83a1e462e9320cad79a3ad791, user.id=f8364b1a190d4218bf353a89bdb81448) {{(pid=81437) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:54:31.694823 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52deae3e-5d5d-43a3-b349-207b060a05c2 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:31.695416 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d7981ba-2b06-4c7c-9465-c0121e11ebdf None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:31.697213 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52deae3e-5d5d-43a3-b349-207b060a05c2 None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:31.704722 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 387/771] 10.222.0.50 () {56 vars in 1374 bytes} [Tue Sep 13 15:54:31 2022] GET /identity/v3/role_assignments?user.id=f8364b1a190d4218bf353a89bdb81448&scope.project.id=5811e3b83a1e462e9320cad79a3ad791&role.id=da847124d9374ac88b27968451c486ab => generated 259 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 13 15:54:31.712371 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:54:31.751402 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fb921ac-92c4-4e49-a073-ab7cd2749a80 None None] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:31.751720 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fb921ac-92c4-4e49-a073-ab7cd2749a80 None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:31.751971 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fb921ac-92c4-4e49-a073-ab7cd2749a80 None None] PATH_INFO: `/` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:31.753772 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 388/772] 10.222.0.50 () {54 vars in 927 bytes} [Tue Sep 13 15:54:31 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 13 15:54:31.759685 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62c54120-9f96-4b84-9881-b78a88fef285 None None] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:31.759994 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62c54120-9f96-4b84-9881-b78a88fef285 None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:31.760239 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62c54120-9f96-4b84-9881-b78a88fef285 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:31.795254 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.auth.core [None req-62c54120-9f96-4b84-9881-b78a88fef285 None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81437) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:54:31.796481 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-62c54120-9f96-4b84-9881-b78a88fef285 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:54:31.812256 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d7981ba-2b06-4c7c-9465-c0121e11ebdf None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:31.864773 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:54:31.888979 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 389/773] 10.222.0.50 () {58 vars in 1024 bytes} [Tue Sep 13 15:54:31 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 131 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 13 15:54:31.891459 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-653bfc30-13e1-45c4-8249-06c1bf4c8a30 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:31.892214 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-653bfc30-13e1-45c4-8249-06c1bf4c8a30 None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:31.892605 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-653bfc30-13e1-45c4-8249-06c1bf4c8a30 None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:31.892983 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-653bfc30-13e1-45c4-8249-06c1bf4c8a30 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:31.893846 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-653bfc30-13e1-45c4-8249-06c1bf4c8a30 None admin] RBAC: Adding query filter params (interface=public, service_id=160b30db2c6545d0af72a3e0802f3d68, region_id=RegionOne) {{(pid=81438) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:54:31.894265 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-653bfc30-13e1-45c4-8249-06c1bf4c8a30 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:31.895860 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-653bfc30-13e1-45c4-8249-06c1bf4c8a30 None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:31.905642 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 385/774] 10.222.0.50 () {56 vars in 1251 bytes} [Tue Sep 13 15:54:31 2022] GET /identity/v3/endpoints?service_id=160b30db2c6545d0af72a3e0802f3d68&interface=public®ion_id=RegionOne => generated 194 bytes in 211 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:54:31.912410 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62c54120-9f96-4b84-9881-b78a88fef285 None None] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:32.029901 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62c54120-9f96-4b84-9881-b78a88fef285 None None] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:32.066572 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7dbce5f5-924d-47af-b349-d15066281770 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:32.067296 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dbce5f5-924d-47af-b349-d15066281770 None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:32.067880 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dbce5f5-924d-47af-b349-d15066281770 None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:32.068156 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dbce5f5-924d-47af-b349-d15066281770 None admin] PATH_INFO: `/v3/services` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:32.068770 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7dbce5f5-924d-47af-b349-d15066281770 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:32.070259 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7dbce5f5-924d-47af-b349-d15066281770 None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:32.076668 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 390/775] 10.222.0.50 () {56 vars in 1087 bytes} [Tue Sep 13 15:54:31 2022] GET /identity/v3/services => generated 772 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:54:32.347808 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-653bfc30-13e1-45c4-8249-06c1bf4c8a30 None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:32.378744 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58ade404-6dcb-459a-9a93-814c36e502cd None None] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:32.379123 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58ade404-6dcb-459a-9a93-814c36e502cd None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:32.379333 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58ade404-6dcb-459a-9a93-814c36e502cd None None] PATH_INFO: `/` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:32.380926 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 391/776] 10.222.0.50 () {54 vars in 927 bytes} [Tue Sep 13 15:54:32 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:54:32.386631 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9ad1a82-5692-4852-8885-7bc837831013 None None] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:32.386954 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9ad1a82-5692-4852-8885-7bc837831013 None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:32.387227 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9ad1a82-5692-4852-8885-7bc837831013 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:32.420640 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.auth.core [None req-f9ad1a82-5692-4852-8885-7bc837831013 None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81437) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:54:32.422055 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-f9ad1a82-5692-4852-8885-7bc837831013 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:54:32.443300 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:54:32.480458 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-653bfc30-13e1-45c4-8249-06c1bf4c8a30 None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:32.486278 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:54:32.514853 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 392/777] 10.222.0.50 () {58 vars in 1024 bytes} [Tue Sep 13 15:54:32 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 129 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 13 15:54:32.522463 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ef38e9c-6bf8-473d-94a3-40d2993448c8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:32.527015 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ef38e9c-6bf8-473d-94a3-40d2993448c8 None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:32.527015 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ef38e9c-6bf8-473d-94a3-40d2993448c8 None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:32.527015 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ef38e9c-6bf8-473d-94a3-40d2993448c8 None admin] PATH_INFO: `/v3/services/network` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:32.527015 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ef38e9c-6bf8-473d-94a3-40d2993448c8 None admin] RBAC: Authorizing `identity:get_service(service_id=network)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:32.527015 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ef38e9c-6bf8-473d-94a3-40d2993448c8 None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:32.536779 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application [None req-2ef38e9c-6bf8-473d-94a3-40d2993448c8 None admin] Could not find service: network.: keystone.exception.ServiceNotFound: Could not find service: network. Sep 13 15:54:32.536779 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:54:32.536779 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 13 15:54:32.536779 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 13 15:54:32.536779 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 13 15:54:32.536779 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 13 15:54:32.536779 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 13 15:54:32.536779 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 13 15:54:32.536779 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Sep 13 15:54:32.536779 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Sep 13 15:54:32.536779 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 13 15:54:32.536779 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Sep 13 15:54:32.536779 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:54:32.536779 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:54:32.536779 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:54:32.536779 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:54:32.536779 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:54:32.537887 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:54:32.537887 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:54:32.537887 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:54:32.537887 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:54:32.537887 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:54:32.537887 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:54:32.537887 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 13 15:54:32.537887 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 13 15:54:32.537887 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 13 15:54:32.537887 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 13 15:54:32.537887 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:54:32.537887 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:54:32.537887 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:54:32.537887 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:54:32.537887 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:54:32.537887 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:54:32.538727 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:54:32.538727 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application with Lock( Sep 13 15:54:32.538727 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:54:32.538727 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:54:32.538727 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:54:32.538727 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:54:32.538727 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:54:32.538727 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:54:32.538727 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:54:32.538727 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:54:32.538727 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 13 15:54:32.538727 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 13 15:54:32.538727 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Sep 13 15:54:32.538727 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Sep 13 15:54:32.542427 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 386/778] 10.222.0.50 () {56 vars in 1111 bytes} [Tue Sep 13 15:54:32 2022] GET /identity/v3/services/network => generated 88 bytes in 195 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:54:32.548285 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9ad1a82-5692-4852-8885-7bc837831013 None None] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:32.655765 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9ad1a82-5692-4852-8885-7bc837831013 None None] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:32.709466 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17a87631-13f7-4d67-ab6a-37220e9c4242 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:32.710376 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17a87631-13f7-4d67-ab6a-37220e9c4242 None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:32.710852 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17a87631-13f7-4d67-ab6a-37220e9c4242 None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:32.711324 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17a87631-13f7-4d67-ab6a-37220e9c4242 None admin] PATH_INFO: `/v3/services` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:32.713407 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17a87631-13f7-4d67-ab6a-37220e9c4242 None admin] RBAC: Adding query filter params (name=network) {{(pid=81437) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:54:32.713887 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17a87631-13f7-4d67-ab6a-37220e9c4242 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:32.715486 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17a87631-13f7-4d67-ab6a-37220e9c4242 None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:32.728390 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 393/779] 10.222.0.50 () {56 vars in 1112 bytes} [Tue Sep 13 15:54:32 2022] GET /identity/v3/services?name=network => generated 124 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:54:32.734277 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ef38e9c-6bf8-473d-94a3-40d2993448c8 None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:32.879676 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ef38e9c-6bf8-473d-94a3-40d2993448c8 None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:32.922221 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bf5c5b3-05af-48e2-85a6-a46f0873b34c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:32.922754 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bf5c5b3-05af-48e2-85a6-a46f0873b34c None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:32.923074 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bf5c5b3-05af-48e2-85a6-a46f0873b34c None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:32.923350 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bf5c5b3-05af-48e2-85a6-a46f0873b34c None admin] PATH_INFO: `/v3/services` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:32.923981 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4bf5c5b3-05af-48e2-85a6-a46f0873b34c None admin] RBAC: Adding query filter params (type=network) {{(pid=81438) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:54:32.924282 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4bf5c5b3-05af-48e2-85a6-a46f0873b34c None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:32.925651 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4bf5c5b3-05af-48e2-85a6-a46f0873b34c None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:32.926930 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:54:32.932099 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 387/780] 10.222.0.50 () {56 vars in 1112 bytes} [Tue Sep 13 15:54:32 2022] GET /identity/v3/services?type=network => generated 350 bytes in 198 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:54:32.938750 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17a87631-13f7-4d67-ab6a-37220e9c4242 None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:33.014351 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:54:33.105948 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17a87631-13f7-4d67-ab6a-37220e9c4242 None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:33.135358 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bf5c5b3-05af-48e2-85a6-a46f0873b34c None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:33.151054 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edb92c7c-b372-46fc-8230-6b43c76258d9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:33.152370 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edb92c7c-b372-46fc-8230-6b43c76258d9 None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:33.152855 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edb92c7c-b372-46fc-8230-6b43c76258d9 None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:33.153354 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edb92c7c-b372-46fc-8230-6b43c76258d9 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:33.154429 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-edb92c7c-b372-46fc-8230-6b43c76258d9 None admin] RBAC: Adding query filter params (interface=public, service_id=c2589ab50acf4aee963f29957fc697b0, region_id=RegionOne) {{(pid=81437) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:54:33.155023 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-edb92c7c-b372-46fc-8230-6b43c76258d9 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:33.156587 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-edb92c7c-b372-46fc-8230-6b43c76258d9 None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:33.167287 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 394/781] 10.222.0.50 () {56 vars in 1251 bytes} [Tue Sep 13 15:54:32 2022] GET /identity/v3/endpoints?service_id=c2589ab50acf4aee963f29957fc697b0&interface=public®ion_id=RegionOne => generated 194 bytes in 229 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:54:33.173050 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edb92c7c-b372-46fc-8230-6b43c76258d9 None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:33.261701 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bf5c5b3-05af-48e2-85a6-a46f0873b34c None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:33.297091 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aecf8d48-04c9-4262-99ee-d574ee78bc6a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:33.297730 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aecf8d48-04c9-4262-99ee-d574ee78bc6a None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:33.298015 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aecf8d48-04c9-4262-99ee-d574ee78bc6a None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:33.298291 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aecf8d48-04c9-4262-99ee-d574ee78bc6a None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:33.309871 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aecf8d48-04c9-4262-99ee-d574ee78bc6a None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 13 15:54:33.315241 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application [None req-aecf8d48-04c9-4262-99ee-d574ee78bc6a None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 13 15:54:33.315241 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:54:33.315241 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:54:33.315241 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:54:33.315241 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:54:33.315241 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:54:33.315241 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:54:33.315241 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:54:33.315241 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:54:33.315241 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:54:33.315241 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:54:33.315241 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:54:33.315241 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 13 15:54:33.315241 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 13 15:54:33.315241 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 13 15:54:33.315241 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 13 15:54:33.316178 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 13 15:54:33.316178 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 13 15:54:33.316178 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 13 15:54:33.316178 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 13 15:54:33.316178 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:54:33.316178 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:54:33.316178 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:54:33.316178 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:54:33.316178 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:54:33.316178 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:54:33.316178 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:54:33.316178 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application with Lock( Sep 13 15:54:33.316178 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:54:33.316178 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:54:33.316178 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:54:33.316178 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:54:33.317061 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:54:33.317061 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:54:33.317061 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:54:33.317061 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:54:33.317061 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 13 15:54:33.317061 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 13 15:54:33.317061 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 13 15:54:33.317061 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 13 15:54:33.317061 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 13 15:54:33.317061 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 13 15:54:33.317061 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 13 15:54:33.317061 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Sep 13 15:54:33.317061 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 388/782] 10.222.0.50 () {56 vars in 1096 bytes} [Tue Sep 13 15:54:33 2022] GET /identity/v3/roles/admin => generated 83 bytes in 182 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 13 15:54:33.324357 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aecf8d48-04c9-4262-99ee-d574ee78bc6a None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:33.393230 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edb92c7c-b372-46fc-8230-6b43c76258d9 None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:33.406842 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:54:33.443387 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:54:33.467975 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2eaf9a45-27e2-4470-a26c-89e914ca6ba9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:33.468591 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2eaf9a45-27e2-4470-a26c-89e914ca6ba9 None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:33.468868 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2eaf9a45-27e2-4470-a26c-89e914ca6ba9 None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:33.469141 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2eaf9a45-27e2-4470-a26c-89e914ca6ba9 None admin] PATH_INFO: `/v3/services` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:33.469749 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2eaf9a45-27e2-4470-a26c-89e914ca6ba9 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:33.471158 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2eaf9a45-27e2-4470-a26c-89e914ca6ba9 None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:33.483023 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 395/783] 10.222.0.50 () {56 vars in 1087 bytes} [Tue Sep 13 15:54:33 2022] GET /identity/v3/services => generated 772 bytes in 310 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:54:33.497896 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aecf8d48-04c9-4262-99ee-d574ee78bc6a None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:33.533814 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b01c035b-8996-4bdd-82a1-5a44feb7812f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:33.534433 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b01c035b-8996-4bdd-82a1-5a44feb7812f None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:33.534710 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b01c035b-8996-4bdd-82a1-5a44feb7812f None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:33.535010 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b01c035b-8996-4bdd-82a1-5a44feb7812f None admin] PATH_INFO: `/v3/roles` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:33.535676 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b01c035b-8996-4bdd-82a1-5a44feb7812f None admin] RBAC: Adding query filter params (name=admin) {{(pid=81438) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:54:33.535978 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b01c035b-8996-4bdd-82a1-5a44feb7812f None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:33.537335 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b01c035b-8996-4bdd-82a1-5a44feb7812f None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:33.546951 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 389/784] 10.222.0.50 () {56 vars in 1099 bytes} [Tue Sep 13 15:54:33 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 223 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 13 15:54:33.556829 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2eaf9a45-27e2-4470-a26c-89e914ca6ba9 None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:33.706017 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2eaf9a45-27e2-4470-a26c-89e914ca6ba9 None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:33.740768 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-061460d0-fb4e-43ec-8141-66c226a18267 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:33.741760 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-061460d0-fb4e-43ec-8141-66c226a18267 None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:33.742241 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-061460d0-fb4e-43ec-8141-66c226a18267 None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:33.742725 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-061460d0-fb4e-43ec-8141-66c226a18267 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:33.757640 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-061460d0-fb4e-43ec-8141-66c226a18267 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:33.759475 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-061460d0-fb4e-43ec-8141-66c226a18267 None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:33.774399 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application [None req-061460d0-fb4e-43ec-8141-66c226a18267 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:54:33.774399 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:54:33.774399 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:54:33.774399 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:54:33.774399 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:54:33.774399 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:54:33.774399 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:54:33.774399 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:54:33.774399 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:54:33.774399 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:54:33.774399 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:54:33.774399 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:54:33.774399 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 13 15:54:33.774399 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 13 15:54:33.774399 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 13 15:54:33.774399 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 13 15:54:33.775516 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:54:33.775516 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:54:33.775516 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:54:33.775516 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:54:33.775516 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:54:33.775516 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:54:33.775516 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:54:33.775516 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application with Lock( Sep 13 15:54:33.775516 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:54:33.775516 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:54:33.775516 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:54:33.775516 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:54:33.775516 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:54:33.775516 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:54:33.775516 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:54:33.775516 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:54:33.775516 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 13 15:54:33.776412 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 13 15:54:33.776412 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:54:33.776412 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application Sep 13 15:54:33.778474 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 396/785] 10.222.0.50 () {56 vars in 1108 bytes} [Tue Sep 13 15:54:33 2022] GET /identity/v3/domains/Default => generated 87 bytes in 223 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:54:33.786402 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b01c035b-8996-4bdd-82a1-5a44feb7812f None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:33.900428 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b01c035b-8996-4bdd-82a1-5a44feb7812f None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:33.923860 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:54:33.946708 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-021e3dd7-5d2b-4dba-97fa-b9cc89fbdf45 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:33.947685 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-021e3dd7-5d2b-4dba-97fa-b9cc89fbdf45 None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:33.948200 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-021e3dd7-5d2b-4dba-97fa-b9cc89fbdf45 None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:33.948669 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-021e3dd7-5d2b-4dba-97fa-b9cc89fbdf45 None admin] PATH_INFO: `/v3/domains` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:33.949507 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-021e3dd7-5d2b-4dba-97fa-b9cc89fbdf45 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81438) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:54:33.950003 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-021e3dd7-5d2b-4dba-97fa-b9cc89fbdf45 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:33.951573 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-021e3dd7-5d2b-4dba-97fa-b9cc89fbdf45 None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:33.953554 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bb9d343-68ee-41b6-9406-968f08a7a9c1 None None] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:33.954366 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bb9d343-68ee-41b6-9406-968f08a7a9c1 None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:33.960520 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bb9d343-68ee-41b6-9406-968f08a7a9c1 None None] PATH_INFO: `/` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:33.963102 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 397/786] 10.222.0.50 () {54 vars in 927 bytes} [Tue Sep 13 15:54:33 2022] GET /identity => generated 270 bytes in 10 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 13 15:54:33.968597 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2630f9a-40d3-465e-abbc-c28e6ec03765 None None] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:33.968892 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2630f9a-40d3-465e-abbc-c28e6ec03765 None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:33.969175 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2630f9a-40d3-465e-abbc-c28e6ec03765 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:33.976955 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 390/787] 10.222.0.50 () {56 vars in 1109 bytes} [Tue Sep 13 15:54:33 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 191 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:54:33.983355 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-021e3dd7-5d2b-4dba-97fa-b9cc89fbdf45 None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:34.005717 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.auth.core [None req-e2630f9a-40d3-465e-abbc-c28e6ec03765 None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81437) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:54:34.008773 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-e2630f9a-40d3-465e-abbc-c28e6ec03765 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:54:34.047257 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:54:34.153090 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 398/788] 10.222.0.50 () {58 vars in 1024 bytes} [Tue Sep 13 15:54:33 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 185 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 13 15:54:34.163620 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-021e3dd7-5d2b-4dba-97fa-b9cc89fbdf45 None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:34.212985 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05564e38-35dc-44dc-ad04-af18bed7ad3b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:34.213548 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05564e38-35dc-44dc-ad04-af18bed7ad3b None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:34.213832 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05564e38-35dc-44dc-ad04-af18bed7ad3b None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:34.214104 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05564e38-35dc-44dc-ad04-af18bed7ad3b None admin] PATH_INFO: `/v3/users/swift` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:34.224787 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05564e38-35dc-44dc-ad04-af18bed7ad3b None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:34.226248 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05564e38-35dc-44dc-ad04-af18bed7ad3b None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:34.238397 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application [None req-05564e38-35dc-44dc-ad04-af18bed7ad3b None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Sep 13 15:54:34.238397 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:54:34.238397 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:54:34.238397 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:54:34.238397 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:54:34.238397 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:54:34.238397 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:54:34.238397 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:54:34.238397 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:54:34.238397 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:54:34.238397 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:54:34.238397 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:54:34.238397 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 13 15:54:34.238397 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 13 15:54:34.238397 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 13 15:54:34.238397 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 13 15:54:34.239410 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:54:34.239410 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:54:34.239410 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 13 15:54:34.239410 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 13 15:54:34.239410 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 13 15:54:34.239410 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 13 15:54:34.239410 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:54:34.239410 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:54:34.239410 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:54:34.239410 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:54:34.239410 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:54:34.239410 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application with Lock( Sep 13 15:54:34.239410 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:54:34.239410 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:54:34.239410 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:54:34.239410 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:54:34.239410 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:54:34.239410 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:54:34.240396 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:54:34.240396 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:54:34.240396 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 13 15:54:34.240396 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 13 15:54:34.240396 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 13 15:54:34.240396 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 13 15:54:34.240396 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 13 15:54:34.240396 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 13 15:54:34.240396 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Sep 13 15:54:34.240396 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Sep 13 15:54:34.244186 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 391/789] 10.222.0.50 () {56 vars in 1096 bytes} [Tue Sep 13 15:54:33 2022] GET /identity/v3/users/swift => generated 83 bytes in 261 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:54:34.249616 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2630f9a-40d3-465e-abbc-c28e6ec03765 None None] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:34.395783 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2630f9a-40d3-465e-abbc-c28e6ec03765 None None] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:34.447680 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a1e684b-006f-4412-9f29-d4d90665f63c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:34.448304 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a1e684b-006f-4412-9f29-d4d90665f63c None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:34.448584 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a1e684b-006f-4412-9f29-d4d90665f63c None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:34.448858 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a1e684b-006f-4412-9f29-d4d90665f63c None admin] PATH_INFO: `/v3/users` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:34.449591 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a1e684b-006f-4412-9f29-d4d90665f63c None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=81437) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:54:34.449897 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a1e684b-006f-4412-9f29-d4d90665f63c None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:34.451391 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a1e684b-006f-4412-9f29-d4d90665f63c None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:34.474125 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 399/790] 10.222.0.50 () {56 vars in 1135 bytes} [Tue Sep 13 15:54:34 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 225 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:54:34.480563 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05564e38-35dc-44dc-ad04-af18bed7ad3b None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:34.546103 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:54:34.597704 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe3cc2b5-0af3-4b61-8902-2915bbd4a778 None None] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:34.598027 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe3cc2b5-0af3-4b61-8902-2915bbd4a778 None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:34.598270 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe3cc2b5-0af3-4b61-8902-2915bbd4a778 None None] PATH_INFO: `/` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:34.599889 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 400/791] 10.222.0.50 () {54 vars in 927 bytes} [Tue Sep 13 15:54:34 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:54:34.606433 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b017d70d-6510-4344-9290-abb5ebb69580 None None] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:34.606714 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b017d70d-6510-4344-9290-abb5ebb69580 None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:34.607028 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b017d70d-6510-4344-9290-abb5ebb69580 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:34.622306 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05564e38-35dc-44dc-ad04-af18bed7ad3b None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:34.647267 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.auth.core [None req-b017d70d-6510-4344-9290-abb5ebb69580 None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81437) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:54:34.648458 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-b017d70d-6510-4344-9290-abb5ebb69580 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:54:34.652029 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:54:34.659937 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a41fa0b-71e1-4fc3-a042-8c6618f5c1c6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:34.660823 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a41fa0b-71e1-4fc3-a042-8c6618f5c1c6 None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:34.661286 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a41fa0b-71e1-4fc3-a042-8c6618f5c1c6 None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:34.661744 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a41fa0b-71e1-4fc3-a042-8c6618f5c1c6 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:34.673644 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a41fa0b-71e1-4fc3-a042-8c6618f5c1c6 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:34.675517 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a41fa0b-71e1-4fc3-a042-8c6618f5c1c6 None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:34.692481 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application [None req-2a41fa0b-71e1-4fc3-a042-8c6618f5c1c6 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:54:34.692481 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:54:34.692481 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:54:34.692481 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:54:34.692481 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:54:34.692481 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:54:34.692481 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:54:34.692481 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:54:34.692481 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:54:34.692481 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:54:34.692481 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:54:34.692481 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:54:34.692481 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 13 15:54:34.692481 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 13 15:54:34.692481 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 13 15:54:34.692481 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 13 15:54:34.693479 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:54:34.693479 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:54:34.693479 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:54:34.693479 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:54:34.693479 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:54:34.693479 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:54:34.693479 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:54:34.693479 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application with Lock( Sep 13 15:54:34.693479 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:54:34.693479 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:54:34.693479 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:54:34.693479 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:54:34.693479 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:54:34.693479 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:54:34.693479 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:54:34.693479 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:54:34.693479 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 13 15:54:34.694394 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 13 15:54:34.694394 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:54:34.694394 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Sep 13 15:54:34.696551 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 392/792] 10.222.0.50 () {56 vars in 1108 bytes} [Tue Sep 13 15:54:34 2022] GET /identity/v3/domains/Default => generated 87 bytes in 216 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:54:34.697533 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a41fa0b-71e1-4fc3-a042-8c6618f5c1c6 None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:34.733471 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 401/793] 10.222.0.50 () {58 vars in 1024 bytes} [Tue Sep 13 15:54:34 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 127 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 13 15:54:34.734081 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b017d70d-6510-4344-9290-abb5ebb69580 None None] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:34.824298 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:54:34.841687 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a41fa0b-71e1-4fc3-a042-8c6618f5c1c6 None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:34.879958 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41382731-0590-41ce-b79c-964372020822 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:34.880690 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41382731-0590-41ce-b79c-964372020822 None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:34.881048 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41382731-0590-41ce-b79c-964372020822 None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:34.881401 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41382731-0590-41ce-b79c-964372020822 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:34.892854 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41382731-0590-41ce-b79c-964372020822 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 13 15:54:34.893010 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b017d70d-6510-4344-9290-abb5ebb69580 None None] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:34.898274 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application [None req-41382731-0590-41ce-b79c-964372020822 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 13 15:54:34.898274 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:54:34.898274 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:54:34.898274 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:54:34.898274 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:54:34.898274 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:54:34.898274 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:54:34.898274 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:54:34.898274 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:54:34.898274 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:54:34.898274 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:54:34.898274 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:54:34.898274 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 13 15:54:34.898274 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 13 15:54:34.898274 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 13 15:54:34.898274 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 13 15:54:34.899751 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 13 15:54:34.899751 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 13 15:54:34.899751 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 13 15:54:34.899751 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 13 15:54:34.899751 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:54:34.899751 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:54:34.899751 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:54:34.899751 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:54:34.899751 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:54:34.899751 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:54:34.899751 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:54:34.899751 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application with Lock( Sep 13 15:54:34.899751 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:54:34.899751 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:54:34.899751 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:54:34.899751 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:54:34.901513 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:54:34.901513 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:54:34.901513 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:54:34.901513 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:54:34.901513 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 13 15:54:34.901513 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 13 15:54:34.901513 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 13 15:54:34.901513 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 13 15:54:34.901513 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 13 15:54:34.901513 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 13 15:54:34.901513 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 13 15:54:34.901513 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Sep 13 15:54:34.904897 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 393/794] 10.222.0.50 () {56 vars in 1096 bytes} [Tue Sep 13 15:54:34 2022] GET /identity/v3/roles/admin => generated 83 bytes in 207 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:54:34.906173 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2473e25f-ce5b-4676-8eeb-7f7cb4d8523f None None] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:34.906459 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2473e25f-ce5b-4676-8eeb-7f7cb4d8523f None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:34.906762 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2473e25f-ce5b-4676-8eeb-7f7cb4d8523f None None] PATH_INFO: `/` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:34.912640 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 394/795] 10.222.0.50 () {54 vars in 927 bytes} [Tue Sep 13 15:54:34 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:54:34.913559 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2473e25f-ce5b-4676-8eeb-7f7cb4d8523f None None] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:34.935277 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88aec6ab-ac77-4a68-adc9-ab0c23395518 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:34.936160 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88aec6ab-ac77-4a68-adc9-ab0c23395518 None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:34.936445 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88aec6ab-ac77-4a68-adc9-ab0c23395518 None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:34.936958 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88aec6ab-ac77-4a68-adc9-ab0c23395518 None admin] PATH_INFO: `/v3/domains` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:34.937977 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88aec6ab-ac77-4a68-adc9-ab0c23395518 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81437) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:54:34.938296 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88aec6ab-ac77-4a68-adc9-ab0c23395518 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:34.939718 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88aec6ab-ac77-4a68-adc9-ab0c23395518 None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:34.952282 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:54:34.957993 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 402/796] 10.222.0.50 () {56 vars in 1109 bytes} [Tue Sep 13 15:54:34 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 224 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:54:34.963312 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de238b7e-cfed-4794-993f-5213debec957 None None] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:34.963606 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de238b7e-cfed-4794-993f-5213debec957 None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:34.963885 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de238b7e-cfed-4794-993f-5213debec957 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:34.982728 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.auth.core [None req-de238b7e-cfed-4794-993f-5213debec957 None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81437) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:54:34.984009 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-de238b7e-cfed-4794-993f-5213debec957 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:54:35.038294 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2473e25f-ce5b-4676-8eeb-7f7cb4d8523f None None] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:35.067125 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 403/797] 10.222.0.50 () {58 vars in 1024 bytes} [Tue Sep 13 15:54:34 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 13 15:54:35.067996 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de238b7e-cfed-4794-993f-5213debec957 None None] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:35.081870 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1203c0a-09aa-4f56-82a8-32619d4f3494 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:35.082816 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1203c0a-09aa-4f56-82a8-32619d4f3494 None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:35.083384 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1203c0a-09aa-4f56-82a8-32619d4f3494 None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:35.083863 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1203c0a-09aa-4f56-82a8-32619d4f3494 None admin] PATH_INFO: `/v3/roles` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:35.084750 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1203c0a-09aa-4f56-82a8-32619d4f3494 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81438) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:54:35.085271 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1203c0a-09aa-4f56-82a8-32619d4f3494 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:35.087634 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1203c0a-09aa-4f56-82a8-32619d4f3494 None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:35.102697 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 395/798] 10.222.0.50 () {56 vars in 1099 bytes} [Tue Sep 13 15:54:34 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:54:35.104166 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1203c0a-09aa-4f56-82a8-32619d4f3494 None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:35.125311 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:54:35.201635 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de238b7e-cfed-4794-993f-5213debec957 None None] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:35.213934 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1203c0a-09aa-4f56-82a8-32619d4f3494 None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:35.241096 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:54:35.250523 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acfb2c5e-fa85-4b00-9af8-c0eb6ea475d6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:35.250681 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adb89fd3-b0d4-4d5f-b082-7a7aa1140e69 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:35.251214 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adb89fd3-b0d4-4d5f-b082-7a7aa1140e69 None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:35.251493 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adb89fd3-b0d4-4d5f-b082-7a7aa1140e69 None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:35.251765 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adb89fd3-b0d4-4d5f-b082-7a7aa1140e69 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:35.251982 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acfb2c5e-fa85-4b00-9af8-c0eb6ea475d6 None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:35.252424 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acfb2c5e-fa85-4b00-9af8-c0eb6ea475d6 None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:35.252698 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acfb2c5e-fa85-4b00-9af8-c0eb6ea475d6 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:35.261468 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-acfb2c5e-fa85-4b00-9af8-c0eb6ea475d6 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:35.262963 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-adb89fd3-b0d4-4d5f-b082-7a7aa1140e69 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 13 15:54:35.263220 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-acfb2c5e-fa85-4b00-9af8-c0eb6ea475d6 None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:35.268262 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application [None req-adb89fd3-b0d4-4d5f-b082-7a7aa1140e69 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 13 15:54:35.268262 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:54:35.268262 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:54:35.268262 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:54:35.268262 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:54:35.268262 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:54:35.268262 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:54:35.268262 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:54:35.268262 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:54:35.268262 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:54:35.268262 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:54:35.268262 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:54:35.268262 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 13 15:54:35.268262 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 13 15:54:35.268262 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 13 15:54:35.268262 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 13 15:54:35.269202 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 13 15:54:35.269202 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 13 15:54:35.269202 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 13 15:54:35.269202 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 13 15:54:35.269202 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:54:35.269202 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:54:35.269202 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:54:35.269202 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:54:35.269202 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:54:35.269202 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:54:35.269202 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:54:35.269202 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application with Lock( Sep 13 15:54:35.269202 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:54:35.269202 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:54:35.269202 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:54:35.269202 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:54:35.270052 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:54:35.270052 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:54:35.270052 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:54:35.270052 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:54:35.270052 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 13 15:54:35.270052 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 13 15:54:35.270052 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 13 15:54:35.270052 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 13 15:54:35.270052 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 13 15:54:35.270052 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 13 15:54:35.270052 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 13 15:54:35.270052 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Sep 13 15:54:35.272742 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 396/799] 10.222.0.50 () {56 vars in 1102 bytes} [Tue Sep 13 15:54:35 2022] GET /identity/v3/roles/service => generated 85 bytes in 169 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:54:35.273696 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adb89fd3-b0d4-4d5f-b082-7a7aa1140e69 None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:35.274819 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application [None req-acfb2c5e-fa85-4b00-9af8-c0eb6ea475d6 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 13 15:54:35.274819 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:54:35.274819 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:54:35.274819 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:54:35.274819 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:54:35.274819 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:54:35.274819 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:54:35.274819 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:54:35.274819 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:54:35.274819 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:54:35.274819 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:54:35.274819 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:54:35.274819 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 13 15:54:35.274819 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 13 15:54:35.274819 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 13 15:54:35.275825 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 13 15:54:35.275825 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:54:35.275825 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:54:35.275825 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:54:35.275825 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:54:35.275825 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:54:35.275825 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:54:35.275825 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:54:35.275825 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application with Lock( Sep 13 15:54:35.275825 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:54:35.275825 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:54:35.275825 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:54:35.275825 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:54:35.275825 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:54:35.275825 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:54:35.275825 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:54:35.275825 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:54:35.276747 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 13 15:54:35.276747 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 13 15:54:35.276747 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 13 15:54:35.276747 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 13 15:54:35.276747 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 13 15:54:35.276747 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 13 15:54:35.276747 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 13 15:54:35.276747 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application Sep 13 15:54:35.281093 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 404/800] 10.222.0.50 () {56 vars in 1111 bytes} [Tue Sep 13 15:54:35 2022] GET /identity/v3/projects/service => generated 88 bytes in 208 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:54:35.282973 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acfb2c5e-fa85-4b00-9af8-c0eb6ea475d6 None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:35.359238 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:54:35.387917 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adb89fd3-b0d4-4d5f-b082-7a7aa1140e69 None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:35.402545 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acfb2c5e-fa85-4b00-9af8-c0eb6ea475d6 None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:35.425795 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b671b17a-57b9-4ad1-b331-835c2b5491b0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:35.426455 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b671b17a-57b9-4ad1-b331-835c2b5491b0 None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:35.426927 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b671b17a-57b9-4ad1-b331-835c2b5491b0 None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:35.427038 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b671b17a-57b9-4ad1-b331-835c2b5491b0 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:35.438124 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b671b17a-57b9-4ad1-b331-835c2b5491b0 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:35.438649 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6706022-3f37-45b8-afab-5e56357957a0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:35.439303 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6706022-3f37-45b8-afab-5e56357957a0 None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:35.439591 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6706022-3f37-45b8-afab-5e56357957a0 None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:35.439693 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b671b17a-57b9-4ad1-b331-835c2b5491b0 None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:35.439870 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6706022-3f37-45b8-afab-5e56357957a0 None admin] PATH_INFO: `/v3/roles` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:35.440527 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6706022-3f37-45b8-afab-5e56357957a0 None admin] RBAC: Adding query filter params (name=service) {{(pid=81437) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:54:35.440830 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6706022-3f37-45b8-afab-5e56357957a0 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:35.442189 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6706022-3f37-45b8-afab-5e56357957a0 None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:35.451100 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 405/801] 10.222.0.50 () {56 vars in 1103 bytes} [Tue Sep 13 15:54:35 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:54:35.451833 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6706022-3f37-45b8-afab-5e56357957a0 None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:35.459623 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application [None req-b671b17a-57b9-4ad1-b331-835c2b5491b0 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:54:35.459623 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:54:35.459623 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:54:35.459623 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:54:35.459623 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:54:35.459623 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:54:35.459623 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:54:35.459623 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:54:35.459623 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:54:35.459623 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:54:35.459623 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:54:35.459623 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:54:35.459623 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 13 15:54:35.459623 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 13 15:54:35.459623 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 13 15:54:35.459623 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 13 15:54:35.460580 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:54:35.460580 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:54:35.460580 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:54:35.460580 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:54:35.460580 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:54:35.460580 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:54:35.460580 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:54:35.460580 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application with Lock( Sep 13 15:54:35.460580 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:54:35.460580 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:54:35.460580 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:54:35.460580 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:54:35.460580 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:54:35.460580 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:54:35.460580 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:54:35.460580 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:54:35.460580 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 13 15:54:35.462098 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 13 15:54:35.462098 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:54:35.462098 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Sep 13 15:54:35.463191 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 397/802] 10.222.0.50 () {56 vars in 1108 bytes} [Tue Sep 13 15:54:35 2022] GET /identity/v3/domains/Default => generated 87 bytes in 190 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:54:35.464295 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b671b17a-57b9-4ad1-b331-835c2b5491b0 None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:35.509714 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:54:35.563999 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6706022-3f37-45b8-afab-5e56357957a0 None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:35.572869 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b671b17a-57b9-4ad1-b331-835c2b5491b0 None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:35.598496 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee9b250d-342c-4705-a40a-311d649dcdc1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:35.599130 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee9b250d-342c-4705-a40a-311d649dcdc1 None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:35.599409 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee9b250d-342c-4705-a40a-311d649dcdc1 None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:35.599679 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee9b250d-342c-4705-a40a-311d649dcdc1 None admin] PATH_INFO: `/v3/projects` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:35.600370 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee9b250d-342c-4705-a40a-311d649dcdc1 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81437) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:54:35.600686 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee9b250d-342c-4705-a40a-311d649dcdc1 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:35.600917 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:54:35.602172 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee9b250d-342c-4705-a40a-311d649dcdc1 None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:35.610555 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-552ec6d1-3913-4ceb-a45a-ebe70e787430 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:35.611595 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-552ec6d1-3913-4ceb-a45a-ebe70e787430 None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:35.611595 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-552ec6d1-3913-4ceb-a45a-ebe70e787430 None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:35.611825 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-552ec6d1-3913-4ceb-a45a-ebe70e787430 None admin] PATH_INFO: `/v3/services/block-storage` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:35.612354 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-552ec6d1-3913-4ceb-a45a-ebe70e787430 None admin] RBAC: Authorizing `identity:get_service(service_id=block-storage)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:35.613710 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-552ec6d1-3913-4ceb-a45a-ebe70e787430 None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:35.620416 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 406/803] 10.222.0.50 () {56 vars in 1148 bytes} [Tue Sep 13 15:54:35 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:54:35.620416 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee9b250d-342c-4705-a40a-311d649dcdc1 None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:35.628342 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application [None req-552ec6d1-3913-4ceb-a45a-ebe70e787430 None admin] Could not find service: block-storage.: keystone.exception.ServiceNotFound: Could not find service: block-storage. Sep 13 15:54:35.628342 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:54:35.628342 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 13 15:54:35.628342 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 13 15:54:35.628342 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 13 15:54:35.628342 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 13 15:54:35.628342 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 13 15:54:35.628342 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 13 15:54:35.628342 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Sep 13 15:54:35.628342 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Sep 13 15:54:35.628342 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 13 15:54:35.628342 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Sep 13 15:54:35.628342 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:54:35.628342 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:54:35.628342 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:54:35.628342 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:54:35.628342 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:54:35.629340 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:54:35.629340 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:54:35.629340 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:54:35.629340 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:54:35.629340 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:54:35.629340 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:54:35.629340 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 13 15:54:35.629340 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 13 15:54:35.629340 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 13 15:54:35.629340 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 13 15:54:35.629340 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:54:35.629340 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:54:35.629340 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:54:35.629340 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:54:35.629340 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:54:35.629340 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:54:35.630210 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:54:35.630210 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application with Lock( Sep 13 15:54:35.630210 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:54:35.630210 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:54:35.630210 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:54:35.630210 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:54:35.630210 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:54:35.630210 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:54:35.630210 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:54:35.630210 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:54:35.630210 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 13 15:54:35.630210 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 13 15:54:35.630210 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Sep 13 15:54:35.630210 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Sep 13 15:54:35.635197 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 398/804] 10.222.0.50 () {56 vars in 1129 bytes} [Tue Sep 13 15:54:35 2022] GET /identity/v3/services/block-storage => generated 94 bytes in 171 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:54:35.635563 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-552ec6d1-3913-4ceb-a45a-ebe70e787430 None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:35.728350 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee9b250d-342c-4705-a40a-311d649dcdc1 None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:35.743686 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:54:35.743905 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-552ec6d1-3913-4ceb-a45a-ebe70e787430 None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:35.767447 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-290e212d-e8a5-4acb-b914-3b2e0bd6c755 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:35.768160 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-290e212d-e8a5-4acb-b914-3b2e0bd6c755 None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:35.768440 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-290e212d-e8a5-4acb-b914-3b2e0bd6c755 None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:35.768842 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-290e212d-e8a5-4acb-b914-3b2e0bd6c755 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:35.779518 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f018c720-034a-4b87-bc47-db1e366dadfc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:35.779825 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-290e212d-e8a5-4acb-b914-3b2e0bd6c755 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:35.780122 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f018c720-034a-4b87-bc47-db1e366dadfc None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:35.780401 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f018c720-034a-4b87-bc47-db1e366dadfc None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:35.780675 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f018c720-034a-4b87-bc47-db1e366dadfc None admin] PATH_INFO: `/v3/domains` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:35.781269 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-290e212d-e8a5-4acb-b914-3b2e0bd6c755 None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:35.781380 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f018c720-034a-4b87-bc47-db1e366dadfc None admin] RBAC: Adding query filter params (name=Default) {{(pid=81438) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:54:35.781625 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f018c720-034a-4b87-bc47-db1e366dadfc None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:35.782997 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f018c720-034a-4b87-bc47-db1e366dadfc None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:35.794557 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 399/805] 10.222.0.50 () {56 vars in 1109 bytes} [Tue Sep 13 15:54:35 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:54:35.795373 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f018c720-034a-4b87-bc47-db1e366dadfc None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:35.797245 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application [None req-290e212d-e8a5-4acb-b914-3b2e0bd6c755 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:54:35.797245 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:54:35.797245 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:54:35.797245 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:54:35.797245 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:54:35.797245 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:54:35.797245 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:54:35.797245 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:54:35.797245 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:54:35.797245 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:54:35.797245 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:54:35.797245 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:54:35.797245 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 13 15:54:35.797245 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 13 15:54:35.797245 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 13 15:54:35.797245 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 13 15:54:35.798188 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:54:35.798188 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:54:35.798188 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:54:35.798188 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:54:35.798188 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:54:35.798188 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:54:35.798188 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:54:35.798188 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application with Lock( Sep 13 15:54:35.798188 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:54:35.798188 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:54:35.798188 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:54:35.798188 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:54:35.798188 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:54:35.798188 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:54:35.798188 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:54:35.798188 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:54:35.798188 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 13 15:54:35.799277 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 13 15:54:35.799277 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:54:35.799277 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application Sep 13 15:54:35.800798 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 407/806] 10.222.0.50 () {56 vars in 1108 bytes} [Tue Sep 13 15:54:35 2022] GET /identity/v3/domains/Default => generated 87 bytes in 185 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:54:35.801845 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-290e212d-e8a5-4acb-b914-3b2e0bd6c755 None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:35.854726 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:54:35.905122 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-290e212d-e8a5-4acb-b914-3b2e0bd6c755 None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:35.911883 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f018c720-034a-4b87-bc47-db1e366dadfc None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:35.941335 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a148c0c-46cb-4ba2-aaa4-b86341266f26 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:35.941929 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a148c0c-46cb-4ba2-aaa4-b86341266f26 None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:35.942227 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a148c0c-46cb-4ba2-aaa4-b86341266f26 None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:35.942473 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a148c0c-46cb-4ba2-aaa4-b86341266f26 None admin] PATH_INFO: `/v3/services` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:35.943129 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a148c0c-46cb-4ba2-aaa4-b86341266f26 None admin] RBAC: Adding query filter params (name=block-storage) {{(pid=81437) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:54:35.943412 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a148c0c-46cb-4ba2-aaa4-b86341266f26 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:35.944769 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a148c0c-46cb-4ba2-aaa4-b86341266f26 None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:35.949376 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 408/807] 10.222.0.50 () {56 vars in 1124 bytes} [Tue Sep 13 15:54:35 2022] GET /identity/v3/services?name=block-storage => generated 130 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:54:35.949985 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a148c0c-46cb-4ba2-aaa4-b86341266f26 None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:35.956209 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67700ab8-0225-4fe6-ad45-28f89cc119c0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:35.956886 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67700ab8-0225-4fe6-ad45-28f89cc119c0 None admin] REQUEST_METHOD: `PUT` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:35.957169 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67700ab8-0225-4fe6-ad45-28f89cc119c0 None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:35.957451 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67700ab8-0225-4fe6-ad45-28f89cc119c0 None admin] PATH_INFO: `/v3/projects/5811e3b83a1e462e9320cad79a3ad791/users/b00ab0d6321d483188bc69deaddcdbbc/roles/031d4e1a23e34f71858fc4a6dfbd4ccb` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:35.959427 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67700ab8-0225-4fe6-ad45-28f89cc119c0 None admin] RBAC: Authorizing `identity:create_grant(project_id=5811e3b83a1e462e9320cad79a3ad791, user_id=b00ab0d6321d483188bc69deaddcdbbc, role_id=031d4e1a23e34f71858fc4a6dfbd4ccb)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:35.960869 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67700ab8-0225-4fe6-ad45-28f89cc119c0 None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:35.979011 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 400/808] 10.222.0.50 () {58 vars in 1439 bytes} [Tue Sep 13 15:54:35 2022] PUT /identity/v3/projects/5811e3b83a1e462e9320cad79a3ad791/users/b00ab0d6321d483188bc69deaddcdbbc/roles/031d4e1a23e34f71858fc4a6dfbd4ccb => generated 0 bytes in 181 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:54:35.979011 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67700ab8-0225-4fe6-ad45-28f89cc119c0 None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:35.991772 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:54:36.084767 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a148c0c-46cb-4ba2-aaa4-b86341266f26 None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:36.111385 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67700ab8-0225-4fe6-ad45-28f89cc119c0 None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:36.123539 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:54:36.127594 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75079b47-2b43-437e-969a-6deafcd90302 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:36.128240 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75079b47-2b43-437e-969a-6deafcd90302 None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:36.128517 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75079b47-2b43-437e-969a-6deafcd90302 None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:36.128789 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75079b47-2b43-437e-969a-6deafcd90302 None admin] PATH_INFO: `/v3/users/nova` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:36.138659 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75079b47-2b43-437e-969a-6deafcd90302 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:36.140168 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75079b47-2b43-437e-969a-6deafcd90302 None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:36.153050 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b710424-5a07-469a-a215-4a52c9b1dc53 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:36.153451 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application [None req-75079b47-2b43-437e-969a-6deafcd90302 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Sep 13 15:54:36.153451 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:54:36.153451 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:54:36.153451 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:54:36.153451 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:54:36.153451 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:54:36.153451 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:54:36.153451 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:54:36.153451 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:54:36.153451 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:54:36.153451 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:54:36.153451 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:54:36.153451 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 13 15:54:36.153451 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 13 15:54:36.153451 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 13 15:54:36.153451 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 13 15:54:36.154379 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:54:36.154379 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:54:36.154379 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 13 15:54:36.154379 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 13 15:54:36.154379 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 13 15:54:36.154379 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 13 15:54:36.154379 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:54:36.154379 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:54:36.154379 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:54:36.154379 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:54:36.154379 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:54:36.154379 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application with Lock( Sep 13 15:54:36.154379 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:54:36.154379 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:54:36.154379 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:54:36.154379 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:54:36.154379 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:54:36.154379 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:54:36.156997 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:54:36.156997 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:54:36.156997 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 13 15:54:36.156997 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 13 15:54:36.156997 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 13 15:54:36.156997 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 13 15:54:36.156997 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 13 15:54:36.156997 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 13 15:54:36.156997 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Sep 13 15:54:36.156997 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application Sep 13 15:54:36.157565 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b710424-5a07-469a-a215-4a52c9b1dc53 None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:36.157565 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b710424-5a07-469a-a215-4a52c9b1dc53 None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:36.157565 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b710424-5a07-469a-a215-4a52c9b1dc53 None admin] PATH_INFO: `/v3/domains` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:36.157565 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b710424-5a07-469a-a215-4a52c9b1dc53 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81438) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:54:36.157565 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b710424-5a07-469a-a215-4a52c9b1dc53 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:36.157565 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b710424-5a07-469a-a215-4a52c9b1dc53 None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:36.159886 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 409/809] 10.222.0.50 () {56 vars in 1093 bytes} [Tue Sep 13 15:54:35 2022] GET /identity/v3/users/nova => generated 82 bytes in 209 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:54:36.160003 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75079b47-2b43-437e-969a-6deafcd90302 None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:36.173221 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 401/810] 10.222.0.50 () {56 vars in 1109 bytes} [Tue Sep 13 15:54:35 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 196 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:54:36.174516 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bab7afaf-5ba1-43e9-ba8d-dfd6a3ca7def None None] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:36.174852 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bab7afaf-5ba1-43e9-ba8d-dfd6a3ca7def None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:36.175215 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bab7afaf-5ba1-43e9-ba8d-dfd6a3ca7def None None] PATH_INFO: `/` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:36.176661 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 402/811] 10.222.0.50 () {54 vars in 927 bytes} [Tue Sep 13 15:54:36 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:54:36.177387 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bab7afaf-5ba1-43e9-ba8d-dfd6a3ca7def None None] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:36.261389 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:54:36.276863 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75079b47-2b43-437e-969a-6deafcd90302 None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:36.297305 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bab7afaf-5ba1-43e9-ba8d-dfd6a3ca7def None None] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:36.314006 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-942145eb-b751-4ad8-b0da-abf0e4dab61c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:36.314547 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-942145eb-b751-4ad8-b0da-abf0e4dab61c None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:36.314825 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-942145eb-b751-4ad8-b0da-abf0e4dab61c None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:36.315130 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-942145eb-b751-4ad8-b0da-abf0e4dab61c None admin] PATH_INFO: `/v3/services` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:36.315899 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-942145eb-b751-4ad8-b0da-abf0e4dab61c None admin] RBAC: Adding query filter params (type=block-storage) {{(pid=81437) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:54:36.316193 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-942145eb-b751-4ad8-b0da-abf0e4dab61c None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:36.317698 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-942145eb-b751-4ad8-b0da-abf0e4dab61c None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:36.329206 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 410/812] 10.222.0.50 () {56 vars in 1124 bytes} [Tue Sep 13 15:54:36 2022] GET /identity/v3/services?type=block-storage => generated 367 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:54:36.330130 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-942145eb-b751-4ad8-b0da-abf0e4dab61c None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:36.335013 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1592ce4-0df5-4411-9b50-ce36ec8d38e7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:36.335743 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1592ce4-0df5-4411-9b50-ce36ec8d38e7 None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:36.336119 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1592ce4-0df5-4411-9b50-ce36ec8d38e7 None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:36.336516 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1592ce4-0df5-4411-9b50-ce36ec8d38e7 None admin] PATH_INFO: `/v3/users` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:36.337336 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1592ce4-0df5-4411-9b50-ce36ec8d38e7 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81438) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:54:36.337735 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1592ce4-0df5-4411-9b50-ce36ec8d38e7 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:36.339274 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1592ce4-0df5-4411-9b50-ce36ec8d38e7 None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:36.357725 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 403/813] 10.222.0.50 () {56 vars in 1133 bytes} [Tue Sep 13 15:54:36 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:54:36.357725 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59b33c5b-bd5d-4404-8d3d-e41941ed30c6 None None] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:36.357725 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59b33c5b-bd5d-4404-8d3d-e41941ed30c6 None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:36.357725 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59b33c5b-bd5d-4404-8d3d-e41941ed30c6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:36.378455 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.auth.core [None req-59b33c5b-bd5d-4404-8d3d-e41941ed30c6 None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81438) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:54:36.379697 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-59b33c5b-bd5d-4404-8d3d-e41941ed30c6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:54:36.405106 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:54:36.436876 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-942145eb-b751-4ad8-b0da-abf0e4dab61c None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:36.452696 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 404/814] 10.222.0.50 () {58 vars in 1024 bytes} [Tue Sep 13 15:54:36 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 13 15:54:36.453599 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59b33c5b-bd5d-4404-8d3d-e41941ed30c6 None None] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:36.486092 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63cb4766-fefe-48d9-bd1e-63d82e337aed None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:36.486877 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63cb4766-fefe-48d9-bd1e-63d82e337aed None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:36.487300 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63cb4766-fefe-48d9-bd1e-63d82e337aed None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:36.487672 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63cb4766-fefe-48d9-bd1e-63d82e337aed None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:36.496042 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63cb4766-fefe-48d9-bd1e-63d82e337aed None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:36.497615 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63cb4766-fefe-48d9-bd1e-63d82e337aed None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:36.509685 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application [None req-63cb4766-fefe-48d9-bd1e-63d82e337aed None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Sep 13 15:54:36.509685 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:54:36.509685 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:54:36.509685 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:54:36.509685 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:54:36.509685 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:54:36.509685 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:54:36.509685 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:54:36.509685 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:54:36.509685 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:54:36.509685 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:54:36.509685 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:54:36.509685 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 13 15:54:36.509685 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 13 15:54:36.509685 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 13 15:54:36.509685 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 13 15:54:36.510820 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:54:36.510820 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:54:36.510820 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 13 15:54:36.510820 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 13 15:54:36.510820 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 13 15:54:36.510820 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 13 15:54:36.510820 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:54:36.510820 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:54:36.510820 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:54:36.510820 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:54:36.510820 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:54:36.510820 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application with Lock( Sep 13 15:54:36.510820 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:54:36.510820 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:54:36.510820 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:54:36.510820 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:54:36.510820 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:54:36.510820 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:54:36.512001 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:54:36.512001 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:54:36.512001 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 13 15:54:36.512001 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 13 15:54:36.512001 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 13 15:54:36.512001 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 13 15:54:36.512001 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 13 15:54:36.512001 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 13 15:54:36.512001 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Sep 13 15:54:36.512001 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application Sep 13 15:54:36.518103 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 411/815] 10.222.0.50 () {56 vars in 1117 bytes} [Tue Sep 13 15:54:36 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 184 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:54:36.518103 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63cb4766-fefe-48d9-bd1e-63d82e337aed None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:36.540255 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:54:36.576700 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59b33c5b-bd5d-4404-8d3d-e41941ed30c6 None None] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:36.617147 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08eaa8db-d41b-4846-9009-97f32fe52fa6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:36.618659 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08eaa8db-d41b-4846-9009-97f32fe52fa6 None admin] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:36.619299 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08eaa8db-d41b-4846-9009-97f32fe52fa6 None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:36.619672 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08eaa8db-d41b-4846-9009-97f32fe52fa6 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:36.620437 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08eaa8db-d41b-4846-9009-97f32fe52fa6 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:36.621899 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08eaa8db-d41b-4846-9009-97f32fe52fa6 None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:36.647859 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 405/816] 10.222.0.50 () {60 vars in 1144 bytes} [Tue Sep 13 15:54:36 2022] POST /identity/v3/endpoints => generated 346 bytes in 194 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 15:54:36.648891 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08eaa8db-d41b-4846-9009-97f32fe52fa6 None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:36.676189 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63cb4766-fefe-48d9-bd1e-63d82e337aed None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:36.698280 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:54:36.772303 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0590e732-fb2b-4375-a7d6-388867aaaab5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:36.773332 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0590e732-fb2b-4375-a7d6-388867aaaab5 None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:36.773774 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0590e732-fb2b-4375-a7d6-388867aaaab5 None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:36.774679 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0590e732-fb2b-4375-a7d6-388867aaaab5 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:36.832078 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08eaa8db-d41b-4846-9009-97f32fe52fa6 None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:36.880250 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0590e732-fb2b-4375-a7d6-388867aaaab5 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:36.880945 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0590e732-fb2b-4375-a7d6-388867aaaab5 None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:36.882879 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5741445a-8f25-4495-ae15-312d4a600bde None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:36.884217 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5741445a-8f25-4495-ae15-312d4a600bde None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:36.885058 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5741445a-8f25-4495-ae15-312d4a600bde None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:36.885568 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5741445a-8f25-4495-ae15-312d4a600bde None admin] PATH_INFO: `/v3/users` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:36.886825 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5741445a-8f25-4495-ae15-312d4a600bde None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=81438) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:54:36.887172 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5741445a-8f25-4495-ae15-312d4a600bde None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:36.888644 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5741445a-8f25-4495-ae15-312d4a600bde None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:36.892949 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:54:36.909758 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 406/817] 10.222.0.50 () {56 vars in 1149 bytes} [Tue Sep 13 15:54:36 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 261 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:54:36.910934 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5741445a-8f25-4495-ae15-312d4a600bde None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:37.001960 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 412/818] 10.222.0.50 () {58 vars in 1282 bytes} [Tue Sep 13 15:54:36 2022] GET /identity/v3/auth/tokens => generated 1002 bytes in 487 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 13 15:54:37.002957 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0590e732-fb2b-4375-a7d6-388867aaaab5 None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:37.003761 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:54:37.050473 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5741445a-8f25-4495-ae15-312d4a600bde None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:37.096996 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97ab46a0-1d0b-4e10-88bf-878c02d27576 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:37.097319 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97ab46a0-1d0b-4e10-88bf-878c02d27576 None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:37.097577 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97ab46a0-1d0b-4e10-88bf-878c02d27576 None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:37.097849 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97ab46a0-1d0b-4e10-88bf-878c02d27576 None admin] PATH_INFO: `/v3/services/network` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:37.098504 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97ab46a0-1d0b-4e10-88bf-878c02d27576 None admin] RBAC: Authorizing `identity:get_service(service_id=network)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:37.099979 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97ab46a0-1d0b-4e10-88bf-878c02d27576 None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:37.109841 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application [None req-97ab46a0-1d0b-4e10-88bf-878c02d27576 None admin] Could not find service: network.: keystone.exception.ServiceNotFound: Could not find service: network. Sep 13 15:54:37.109841 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:54:37.109841 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 13 15:54:37.109841 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 13 15:54:37.109841 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 13 15:54:37.109841 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 13 15:54:37.109841 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 13 15:54:37.109841 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 13 15:54:37.109841 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Sep 13 15:54:37.109841 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Sep 13 15:54:37.109841 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 13 15:54:37.109841 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Sep 13 15:54:37.109841 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:54:37.109841 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:54:37.109841 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:54:37.109841 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:54:37.109841 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:54:37.110949 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:54:37.110949 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:54:37.110949 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:54:37.110949 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:54:37.110949 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:54:37.110949 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:54:37.110949 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 13 15:54:37.110949 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 13 15:54:37.110949 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 13 15:54:37.110949 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 13 15:54:37.110949 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:54:37.110949 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:54:37.110949 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:54:37.110949 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:54:37.110949 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:54:37.110949 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:54:37.111819 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:54:37.111819 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application with Lock( Sep 13 15:54:37.111819 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:54:37.111819 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:54:37.111819 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:54:37.111819 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:54:37.111819 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:54:37.111819 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:54:37.111819 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:54:37.111819 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:54:37.111819 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 13 15:54:37.111819 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 13 15:54:37.111819 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Sep 13 15:54:37.111819 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Sep 13 15:54:37.111819 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 407/819] 10.222.0.50 () {56 vars in 1111 bytes} [Tue Sep 13 15:54:36 2022] GET /identity/v3/services/network => generated 88 bytes in 201 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:54:37.113169 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97ab46a0-1d0b-4e10-88bf-878c02d27576 None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:37.135755 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0590e732-fb2b-4375-a7d6-388867aaaab5 None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:37.155293 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:54:37.186624 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b268a62-eeb7-4df5-9b5d-7f7ac67dedb6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:37.187872 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b268a62-eeb7-4df5-9b5d-7f7ac67dedb6 None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:37.188353 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b268a62-eeb7-4df5-9b5d-7f7ac67dedb6 None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:37.188820 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b268a62-eeb7-4df5-9b5d-7f7ac67dedb6 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:37.210650 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b268a62-eeb7-4df5-9b5d-7f7ac67dedb6 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:37.212228 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b268a62-eeb7-4df5-9b5d-7f7ac67dedb6 None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:37.230967 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application [None req-1b268a62-eeb7-4df5-9b5d-7f7ac67dedb6 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:54:37.230967 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:54:37.230967 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:54:37.230967 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:54:37.230967 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:54:37.230967 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:54:37.230967 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:54:37.230967 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:54:37.230967 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:54:37.230967 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:54:37.230967 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:54:37.230967 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:54:37.230967 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 13 15:54:37.230967 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 13 15:54:37.230967 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 13 15:54:37.230967 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 13 15:54:37.232107 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:54:37.232107 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:54:37.232107 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:54:37.232107 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:54:37.232107 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:54:37.232107 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:54:37.232107 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:54:37.232107 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application with Lock( Sep 13 15:54:37.232107 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:54:37.232107 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:54:37.232107 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:54:37.232107 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:54:37.232107 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:54:37.232107 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:54:37.232107 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:54:37.232107 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:54:37.232107 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 13 15:54:37.233607 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 13 15:54:37.233607 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:54:37.233607 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application Sep 13 15:54:37.233607 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 413/820] 10.222.0.50 () {56 vars in 1108 bytes} [Tue Sep 13 15:54:37 2022] GET /identity/v3/domains/Default => generated 87 bytes in 230 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:54:37.233607 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b268a62-eeb7-4df5-9b5d-7f7ac67dedb6 None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:37.234447 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97ab46a0-1d0b-4e10-88bf-878c02d27576 None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:37.277915 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42127ed2-0f6e-44fb-bc95-2cb41729b004 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:37.279493 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42127ed2-0f6e-44fb-bc95-2cb41729b004 None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:37.279843 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42127ed2-0f6e-44fb-bc95-2cb41729b004 None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:37.280122 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42127ed2-0f6e-44fb-bc95-2cb41729b004 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:37.283015 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:54:37.294302 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42127ed2-0f6e-44fb-bc95-2cb41729b004 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:37.296070 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42127ed2-0f6e-44fb-bc95-2cb41729b004 None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:37.316467 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application [None req-42127ed2-0f6e-44fb-bc95-2cb41729b004 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:54:37.316467 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:54:37.316467 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:54:37.316467 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:54:37.316467 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:54:37.316467 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:54:37.316467 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:54:37.316467 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:54:37.316467 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:54:37.316467 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:54:37.316467 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:54:37.316467 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:54:37.316467 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 13 15:54:37.316467 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 13 15:54:37.316467 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 13 15:54:37.316467 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 13 15:54:37.317485 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:54:37.317485 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:54:37.317485 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:54:37.317485 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:54:37.317485 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:54:37.317485 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:54:37.317485 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:54:37.317485 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application with Lock( Sep 13 15:54:37.317485 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:54:37.317485 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:54:37.317485 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:54:37.317485 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:54:37.317485 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:54:37.317485 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:54:37.317485 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:54:37.317485 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:54:37.317485 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 13 15:54:37.318400 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 13 15:54:37.318400 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:54:37.318400 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Sep 13 15:54:37.320538 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 408/821] 10.222.0.50 () {56 vars in 1108 bytes} [Tue Sep 13 15:54:37 2022] GET /identity/v3/domains/Default => generated 87 bytes in 208 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:54:37.322338 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42127ed2-0f6e-44fb-bc95-2cb41729b004 None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:37.369985 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b268a62-eeb7-4df5-9b5d-7f7ac67dedb6 None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:37.418429 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07efe196-3ccf-457d-bbe0-4946e8cec74f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:37.419186 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07efe196-3ccf-457d-bbe0-4946e8cec74f None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:37.419460 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07efe196-3ccf-457d-bbe0-4946e8cec74f None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:37.419732 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07efe196-3ccf-457d-bbe0-4946e8cec74f None admin] PATH_INFO: `/v3/services` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:37.420357 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07efe196-3ccf-457d-bbe0-4946e8cec74f None admin] RBAC: Adding query filter params (name=network) {{(pid=81437) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:54:37.420655 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07efe196-3ccf-457d-bbe0-4946e8cec74f None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:37.422625 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07efe196-3ccf-457d-bbe0-4946e8cec74f None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:37.429561 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 414/822] 10.222.0.50 () {56 vars in 1112 bytes} [Tue Sep 13 15:54:37 2022] GET /identity/v3/services?name=network => generated 124 bytes in 196 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:54:37.430581 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07efe196-3ccf-457d-bbe0-4946e8cec74f None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:37.435931 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42127ed2-0f6e-44fb-bc95-2cb41729b004 None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:37.465117 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:54:37.474307 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a259eee0-a1d5-4674-a86e-84c5ed42effd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:37.475349 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a259eee0-a1d5-4674-a86e-84c5ed42effd None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:37.475841 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a259eee0-a1d5-4674-a86e-84c5ed42effd None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:37.476373 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a259eee0-a1d5-4674-a86e-84c5ed42effd None admin] PATH_INFO: `/v3/domains` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:37.477587 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a259eee0-a1d5-4674-a86e-84c5ed42effd None admin] RBAC: Adding query filter params (name=Default) {{(pid=81438) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:54:37.478195 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a259eee0-a1d5-4674-a86e-84c5ed42effd None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:37.479800 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a259eee0-a1d5-4674-a86e-84c5ed42effd None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:37.493857 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 409/823] 10.222.0.50 () {56 vars in 1109 bytes} [Tue Sep 13 15:54:37 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:54:37.495208 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a259eee0-a1d5-4674-a86e-84c5ed42effd None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:37.550696 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:54:37.563743 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07efe196-3ccf-457d-bbe0-4946e8cec74f None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:37.603509 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6036bba2-814a-40a7-a412-4a48c196b7a5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:37.604099 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6036bba2-814a-40a7-a412-4a48c196b7a5 None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:37.604382 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6036bba2-814a-40a7-a412-4a48c196b7a5 None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:37.604660 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6036bba2-814a-40a7-a412-4a48c196b7a5 None admin] PATH_INFO: `/v3/domains` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:37.605512 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6036bba2-814a-40a7-a412-4a48c196b7a5 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81437) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:54:37.605623 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6036bba2-814a-40a7-a412-4a48c196b7a5 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:37.607006 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6036bba2-814a-40a7-a412-4a48c196b7a5 None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:37.615314 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a259eee0-a1d5-4674-a86e-84c5ed42effd None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:37.619869 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 415/824] 10.222.0.50 () {56 vars in 1109 bytes} [Tue Sep 13 15:54:37 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 190 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:54:37.620769 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6036bba2-814a-40a7-a412-4a48c196b7a5 None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:37.665584 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-caf77453-4729-408f-9ca2-115c89a908d6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:37.666357 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-caf77453-4729-408f-9ca2-115c89a908d6 None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:37.666742 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-caf77453-4729-408f-9ca2-115c89a908d6 None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:37.667142 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-caf77453-4729-408f-9ca2-115c89a908d6 None admin] PATH_INFO: `/v3/services` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:37.667907 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-caf77453-4729-408f-9ca2-115c89a908d6 None admin] RBAC: Adding query filter params (type=network) {{(pid=81438) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:54:37.668314 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-caf77453-4729-408f-9ca2-115c89a908d6 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:37.669822 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-caf77453-4729-408f-9ca2-115c89a908d6 None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:37.676568 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 410/825] 10.222.0.50 () {56 vars in 1112 bytes} [Tue Sep 13 15:54:37 2022] GET /identity/v3/services?type=network => generated 350 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:54:37.677629 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-caf77453-4729-408f-9ca2-115c89a908d6 None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:37.744327 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:54:37.766399 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6036bba2-814a-40a7-a412-4a48c196b7a5 None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:37.788202 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-caf77453-4729-408f-9ca2-115c89a908d6 None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:37.802930 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:54:37.811099 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3569f60-cc5c-41e3-bc3c-db2e5ee5a780 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:37.811741 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3569f60-cc5c-41e3-bc3c-db2e5ee5a780 None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:37.812023 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3569f60-cc5c-41e3-bc3c-db2e5ee5a780 None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:37.812302 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3569f60-cc5c-41e3-bc3c-db2e5ee5a780 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:37.821017 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3569f60-cc5c-41e3-bc3c-db2e5ee5a780 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:37.823100 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3569f60-cc5c-41e3-bc3c-db2e5ee5a780 None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:37.830114 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d59ccf2f-e60c-4715-a1ed-d83929961f20 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:37.831040 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d59ccf2f-e60c-4715-a1ed-d83929961f20 None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:37.831540 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d59ccf2f-e60c-4715-a1ed-d83929961f20 None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:37.832026 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d59ccf2f-e60c-4715-a1ed-d83929961f20 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:37.837435 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application [None req-a3569f60-cc5c-41e3-bc3c-db2e5ee5a780 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 13 15:54:37.837435 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:54:37.837435 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:54:37.837435 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:54:37.837435 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:54:37.837435 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:54:37.837435 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:54:37.837435 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:54:37.837435 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:54:37.837435 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:54:37.837435 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:54:37.837435 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:54:37.837435 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 13 15:54:37.837435 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 13 15:54:37.837435 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 13 15:54:37.838351 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 13 15:54:37.838351 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:54:37.838351 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:54:37.838351 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:54:37.838351 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:54:37.838351 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:54:37.838351 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:54:37.838351 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:54:37.838351 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application with Lock( Sep 13 15:54:37.838351 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:54:37.838351 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:54:37.838351 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:54:37.838351 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:54:37.838351 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:54:37.838351 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:54:37.838351 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:54:37.838351 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:54:37.839335 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 13 15:54:37.839335 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 13 15:54:37.839335 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 13 15:54:37.839335 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 13 15:54:37.839335 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 13 15:54:37.839335 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 13 15:54:37.839335 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 13 15:54:37.839335 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application Sep 13 15:54:37.843386 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 416/826] 10.222.0.50 () {56 vars in 1111 bytes} [Tue Sep 13 15:54:37 2022] GET /identity/v3/projects/service => generated 88 bytes in 218 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:54:37.844370 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3569f60-cc5c-41e3-bc3c-db2e5ee5a780 None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:37.844708 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d59ccf2f-e60c-4715-a1ed-d83929961f20 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:37.846365 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d59ccf2f-e60c-4715-a1ed-d83929961f20 None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:37.859416 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application [None req-d59ccf2f-e60c-4715-a1ed-d83929961f20 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 13 15:54:37.859416 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:54:37.859416 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:54:37.859416 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:54:37.859416 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:54:37.859416 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:54:37.859416 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:54:37.859416 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:54:37.859416 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:54:37.859416 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:54:37.859416 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:54:37.859416 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:54:37.859416 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 13 15:54:37.859416 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 13 15:54:37.859416 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 13 15:54:37.860373 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 13 15:54:37.860373 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:54:37.860373 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:54:37.860373 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:54:37.860373 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:54:37.860373 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:54:37.860373 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:54:37.860373 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:54:37.860373 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application with Lock( Sep 13 15:54:37.860373 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:54:37.860373 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:54:37.860373 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:54:37.860373 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:54:37.860373 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:54:37.860373 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:54:37.860373 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:54:37.860373 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:54:37.861287 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 13 15:54:37.861287 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 13 15:54:37.861287 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 13 15:54:37.861287 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 13 15:54:37.861287 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 13 15:54:37.861287 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 13 15:54:37.861287 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 13 15:54:37.861287 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Sep 13 15:54:37.863990 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 411/827] 10.222.0.50 () {56 vars in 1111 bytes} [Tue Sep 13 15:54:37 2022] GET /identity/v3/projects/service => generated 88 bytes in 187 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:54:37.865204 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d59ccf2f-e60c-4715-a1ed-d83929961f20 None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:37.975212 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3569f60-cc5c-41e3-bc3c-db2e5ee5a780 None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:38.000073 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d59ccf2f-e60c-4715-a1ed-d83929961f20 None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:38.013408 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb4f0d71-9f8a-4f0c-886c-1dd56b2e49c0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:38.015242 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb4f0d71-9f8a-4f0c-886c-1dd56b2e49c0 None admin] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:38.015868 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb4f0d71-9f8a-4f0c-886c-1dd56b2e49c0 None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:38.016363 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb4f0d71-9f8a-4f0c-886c-1dd56b2e49c0 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:38.017384 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb4f0d71-9f8a-4f0c-886c-1dd56b2e49c0 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:38.022542 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb4f0d71-9f8a-4f0c-886c-1dd56b2e49c0 None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:38.046358 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-683338a2-6b3d-4819-8385-228eb98a18b5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:38.047365 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-683338a2-6b3d-4819-8385-228eb98a18b5 None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:38.047724 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-683338a2-6b3d-4819-8385-228eb98a18b5 None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:38.048691 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-683338a2-6b3d-4819-8385-228eb98a18b5 None admin] PATH_INFO: `/v3/projects` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:38.049677 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-683338a2-6b3d-4819-8385-228eb98a18b5 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81438) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:54:38.050153 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-683338a2-6b3d-4819-8385-228eb98a18b5 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:38.052169 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-683338a2-6b3d-4819-8385-228eb98a18b5 None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:38.056817 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:54:38.068219 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 417/828] 10.222.0.50 () {60 vars in 1144 bytes} [Tue Sep 13 15:54:37 2022] POST /identity/v3/endpoints => generated 337 bytes in 224 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 15:54:38.069722 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb4f0d71-9f8a-4f0c-886c-1dd56b2e49c0 None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:38.202874 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 412/829] 10.222.0.50 () {56 vars in 1148 bytes} [Tue Sep 13 15:54:37 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 338 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:54:38.209454 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-683338a2-6b3d-4819-8385-228eb98a18b5 None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:38.280123 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:54:38.305822 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb4f0d71-9f8a-4f0c-886c-1dd56b2e49c0 None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:38.350021 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-731b8aa2-8abe-4a90-b682-e7a8cfc371c5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:38.350642 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-731b8aa2-8abe-4a90-b682-e7a8cfc371c5 None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:38.350979 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-731b8aa2-8abe-4a90-b682-e7a8cfc371c5 None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:38.351215 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-731b8aa2-8abe-4a90-b682-e7a8cfc371c5 None admin] PATH_INFO: `/v3/projects` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:38.352618 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-731b8aa2-8abe-4a90-b682-e7a8cfc371c5 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81437) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:54:38.353103 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-731b8aa2-8abe-4a90-b682-e7a8cfc371c5 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:38.354684 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-731b8aa2-8abe-4a90-b682-e7a8cfc371c5 None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:38.367271 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 418/830] 10.222.0.50 () {56 vars in 1148 bytes} [Tue Sep 13 15:54:38 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 298 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:54:38.368956 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d381d220-625e-4ed5-8ae6-d91793d49f34 None None] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:38.369244 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d381d220-625e-4ed5-8ae6-d91793d49f34 None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:38.369526 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d381d220-625e-4ed5-8ae6-d91793d49f34 None None] PATH_INFO: `/` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:38.371028 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 419/831] 10.222.0.50 () {54 vars in 927 bytes} [Tue Sep 13 15:54:38 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:54:38.375321 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d381d220-625e-4ed5-8ae6-d91793d49f34 None None] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:38.383971 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-683338a2-6b3d-4819-8385-228eb98a18b5 None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:38.424066 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e3cf1a7-9131-41a4-aba3-fc5a07c60f7f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:38.424732 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e3cf1a7-9131-41a4-aba3-fc5a07c60f7f None admin] REQUEST_METHOD: `PUT` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:38.425003 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e3cf1a7-9131-41a4-aba3-fc5a07c60f7f None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:38.425269 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e3cf1a7-9131-41a4-aba3-fc5a07c60f7f None admin] PATH_INFO: `/v3/projects/5811e3b83a1e462e9320cad79a3ad791/users/f8364b1a190d4218bf353a89bdb81448/roles/da847124d9374ac88b27968451c486ab` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:38.440061 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e3cf1a7-9131-41a4-aba3-fc5a07c60f7f None admin] RBAC: Authorizing `identity:create_grant(project_id=5811e3b83a1e462e9320cad79a3ad791, user_id=f8364b1a190d4218bf353a89bdb81448, role_id=da847124d9374ac88b27968451c486ab)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:38.441480 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e3cf1a7-9131-41a4-aba3-fc5a07c60f7f None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:38.447816 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:54:38.458987 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 413/832] 10.222.0.50 () {58 vars in 1439 bytes} [Tue Sep 13 15:54:38 2022] PUT /identity/v3/projects/5811e3b83a1e462e9320cad79a3ad791/users/f8364b1a190d4218bf353a89bdb81448/roles/da847124d9374ac88b27968451c486ab => generated 0 bytes in 248 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:54:38.460953 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c47b159a-84a8-4dfd-be7d-320a151b8f10 None None] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:38.461618 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c47b159a-84a8-4dfd-be7d-320a151b8f10 None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:38.462291 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c47b159a-84a8-4dfd-be7d-320a151b8f10 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:38.487637 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.auth.core [None req-c47b159a-84a8-4dfd-be7d-320a151b8f10 None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81438) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:54:38.489890 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-c47b159a-84a8-4dfd-be7d-320a151b8f10 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:54:38.499287 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d381d220-625e-4ed5-8ae6-d91793d49f34 None None] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:38.539520 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca30c108-e8ad-41ab-bf1e-b46f63145630 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:38.540127 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca30c108-e8ad-41ab-bf1e-b46f63145630 None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:38.540606 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca30c108-e8ad-41ab-bf1e-b46f63145630 None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:38.541028 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca30c108-e8ad-41ab-bf1e-b46f63145630 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:38.542030 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca30c108-e8ad-41ab-bf1e-b46f63145630 None admin] RBAC: Adding query filter params (role.id=031d4e1a23e34f71858fc4a6dfbd4ccb, scope.project.id=5811e3b83a1e462e9320cad79a3ad791, user.id=4e039b46c5e3445bbb39904b00c5430e) {{(pid=81437) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:54:38.542447 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca30c108-e8ad-41ab-bf1e-b46f63145630 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:38.544133 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca30c108-e8ad-41ab-bf1e-b46f63145630 None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:38.551343 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 420/833] 10.222.0.50 () {56 vars in 1374 bytes} [Tue Sep 13 15:54:38 2022] GET /identity/v3/role_assignments?user.id=4e039b46c5e3445bbb39904b00c5430e&scope.project.id=5811e3b83a1e462e9320cad79a3ad791&role.id=031d4e1a23e34f71858fc4a6dfbd4ccb => generated 608 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 13 15:54:38.583063 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 414/834] 10.222.0.50 () {58 vars in 1024 bytes} [Tue Sep 13 15:54:38 2022] POST /identity/v3/auth/tokens => generated 1161 bytes in 123 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 13 15:54:38.860301 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca30c108-e8ad-41ab-bf1e-b46f63145630 None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:38.964732 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca30c108-e8ad-41ab-bf1e-b46f63145630 None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:38.988566 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:54:39.005081 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4f8c897-c6ed-4a78-9afa-9f10778c0898 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:39.005723 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4f8c897-c6ed-4a78-9afa-9f10778c0898 None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:39.006002 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4f8c897-c6ed-4a78-9afa-9f10778c0898 None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:39.006276 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4f8c897-c6ed-4a78-9afa-9f10778c0898 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:39.018022 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4f8c897-c6ed-4a78-9afa-9f10778c0898 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 13 15:54:39.023339 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application [None req-b4f8c897-c6ed-4a78-9afa-9f10778c0898 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 13 15:54:39.023339 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:54:39.023339 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:54:39.023339 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:54:39.023339 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:54:39.023339 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:54:39.023339 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:54:39.023339 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:54:39.023339 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:54:39.023339 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:54:39.023339 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:54:39.023339 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:54:39.023339 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 13 15:54:39.023339 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 13 15:54:39.023339 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 13 15:54:39.023339 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 13 15:54:39.024310 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 13 15:54:39.024310 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 13 15:54:39.024310 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 13 15:54:39.024310 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 13 15:54:39.024310 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:54:39.024310 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:54:39.024310 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:54:39.024310 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:54:39.024310 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:54:39.024310 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:54:39.024310 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:54:39.024310 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application with Lock( Sep 13 15:54:39.024310 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:54:39.024310 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:54:39.024310 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:54:39.024310 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:54:39.025548 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:54:39.025548 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:54:39.025548 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:54:39.025548 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:54:39.025548 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 13 15:54:39.025548 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 13 15:54:39.025548 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 13 15:54:39.025548 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 13 15:54:39.025548 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 13 15:54:39.025548 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 13 15:54:39.025548 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 13 15:54:39.025548 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application Sep 13 15:54:39.025548 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 421/835] 10.222.0.50 () {56 vars in 1096 bytes} [Tue Sep 13 15:54:38 2022] GET /identity/v3/roles/admin => generated 83 bytes in 165 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:54:39.030852 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c47b159a-84a8-4dfd-be7d-320a151b8f10 None None] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:39.048404 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:54:39.114933 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40a98fe7-6b5b-4fb7-b7b5-3c3f4abb5bee None None] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:39.115238 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40a98fe7-6b5b-4fb7-b7b5-3c3f4abb5bee None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:39.115409 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40a98fe7-6b5b-4fb7-b7b5-3c3f4abb5bee None None] PATH_INFO: `/` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:39.117057 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 422/836] 10.222.0.50 () {54 vars in 927 bytes} [Tue Sep 13 15:54:39 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 13 15:54:39.125003 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-363de5ca-e8fb-4f7d-b1e0-1e05a2a74d41 None None] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:39.125482 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-363de5ca-e8fb-4f7d-b1e0-1e05a2a74d41 None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:39.126072 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-363de5ca-e8fb-4f7d-b1e0-1e05a2a74d41 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:39.146139 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c47b159a-84a8-4dfd-be7d-320a151b8f10 None None] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:39.146346 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.auth.core [None req-363de5ca-e8fb-4f7d-b1e0-1e05a2a74d41 None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81437) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:54:39.147408 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-363de5ca-e8fb-4f7d-b1e0-1e05a2a74d41 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:54:39.182717 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0be6823-bc89-4691-9dd6-644debb30d39 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:39.183718 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0be6823-bc89-4691-9dd6-644debb30d39 None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:39.184204 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0be6823-bc89-4691-9dd6-644debb30d39 None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:39.184707 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0be6823-bc89-4691-9dd6-644debb30d39 None admin] PATH_INFO: `/v3/roles` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:39.185568 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0be6823-bc89-4691-9dd6-644debb30d39 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81438) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:54:39.186142 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0be6823-bc89-4691-9dd6-644debb30d39 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:39.187853 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0be6823-bc89-4691-9dd6-644debb30d39 None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:39.198118 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 415/837] 10.222.0.50 () {56 vars in 1099 bytes} [Tue Sep 13 15:54:39 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:54:39.206228 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0be6823-bc89-4691-9dd6-644debb30d39 None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:39.221579 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 423/838] 10.222.0.50 () {58 vars in 1024 bytes} [Tue Sep 13 15:54:39 2022] POST /identity/v3/auth/tokens => generated 1161 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 13 15:54:39.286610 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:54:39.314638 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0be6823-bc89-4691-9dd6-644debb30d39 None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:39.351457 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b6783d1-e8c3-44f7-9f8b-d7c0cbd2304c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:39.352727 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b6783d1-e8c3-44f7-9f8b-d7c0cbd2304c None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:39.353217 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b6783d1-e8c3-44f7-9f8b-d7c0cbd2304c None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:39.353635 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b6783d1-e8c3-44f7-9f8b-d7c0cbd2304c None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:39.365327 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b6783d1-e8c3-44f7-9f8b-d7c0cbd2304c None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:39.366763 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b6783d1-e8c3-44f7-9f8b-d7c0cbd2304c None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:39.381785 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application [None req-2b6783d1-e8c3-44f7-9f8b-d7c0cbd2304c None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:54:39.381785 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:54:39.381785 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:54:39.381785 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:54:39.381785 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:54:39.381785 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:54:39.381785 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:54:39.381785 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:54:39.381785 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:54:39.381785 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:54:39.381785 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:54:39.381785 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:54:39.381785 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 13 15:54:39.381785 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 13 15:54:39.381785 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 13 15:54:39.381785 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 13 15:54:39.382759 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:54:39.382759 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:54:39.382759 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:54:39.382759 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:54:39.382759 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:54:39.382759 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:54:39.382759 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:54:39.382759 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application with Lock( Sep 13 15:54:39.382759 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:54:39.382759 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:54:39.382759 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:54:39.382759 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:54:39.382759 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:54:39.382759 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:54:39.382759 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:54:39.382759 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:54:39.382759 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 13 15:54:39.383912 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 13 15:54:39.383912 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:54:39.383912 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Sep 13 15:54:39.383912 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 416/839] 10.222.0.50 () {56 vars in 1108 bytes} [Tue Sep 13 15:54:39 2022] GET /identity/v3/domains/Default => generated 87 bytes in 180 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:54:39.388776 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-363de5ca-e8fb-4f7d-b1e0-1e05a2a74d41 None None] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:39.496972 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-363de5ca-e8fb-4f7d-b1e0-1e05a2a74d41 None None] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:39.510625 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:54:39.516900 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b6783d1-e8c3-44f7-9f8b-d7c0cbd2304c None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:39.546990 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74937111-1e4f-43f8-89f1-3d13cb113446 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:39.547980 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74937111-1e4f-43f8-89f1-3d13cb113446 None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:39.548466 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74937111-1e4f-43f8-89f1-3d13cb113446 None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:39.548972 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74937111-1e4f-43f8-89f1-3d13cb113446 None admin] PATH_INFO: `/v3/domains` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:39.549839 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74937111-1e4f-43f8-89f1-3d13cb113446 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81437) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:54:39.550347 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74937111-1e4f-43f8-89f1-3d13cb113446 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:39.551982 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74937111-1e4f-43f8-89f1-3d13cb113446 None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:39.564566 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 424/840] 10.222.0.50 () {56 vars in 1109 bytes} [Tue Sep 13 15:54:39 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:54:39.570218 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74937111-1e4f-43f8-89f1-3d13cb113446 None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:39.635114 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b6783d1-e8c3-44f7-9f8b-d7c0cbd2304c None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:39.665650 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:54:39.676376 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c791d832-d63f-455b-9e9d-3fcfcad01add None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:39.677092 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c791d832-d63f-455b-9e9d-3fcfcad01add None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:39.677442 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c791d832-d63f-455b-9e9d-3fcfcad01add None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:39.677786 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c791d832-d63f-455b-9e9d-3fcfcad01add None admin] PATH_INFO: `/v3/services/volumev3` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:39.678367 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c791d832-d63f-455b-9e9d-3fcfcad01add None admin] RBAC: Authorizing `identity:get_service(service_id=volumev3)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:39.680064 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c791d832-d63f-455b-9e9d-3fcfcad01add None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:39.691446 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application [None req-c791d832-d63f-455b-9e9d-3fcfcad01add None admin] Could not find service: volumev3.: keystone.exception.ServiceNotFound: Could not find service: volumev3. Sep 13 15:54:39.691446 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:54:39.691446 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 13 15:54:39.691446 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 13 15:54:39.691446 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 13 15:54:39.691446 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 13 15:54:39.691446 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 13 15:54:39.691446 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 13 15:54:39.691446 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Sep 13 15:54:39.691446 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Sep 13 15:54:39.691446 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 13 15:54:39.691446 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Sep 13 15:54:39.691446 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:54:39.691446 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:54:39.691446 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:54:39.691446 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:54:39.691446 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:54:39.692453 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:54:39.692453 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:54:39.692453 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:54:39.692453 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:54:39.692453 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:54:39.692453 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:54:39.692453 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 13 15:54:39.692453 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 13 15:54:39.692453 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 13 15:54:39.692453 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 13 15:54:39.692453 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:54:39.692453 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:54:39.692453 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:54:39.692453 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:54:39.692453 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:54:39.692453 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:54:39.693378 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:54:39.693378 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application with Lock( Sep 13 15:54:39.693378 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:54:39.693378 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:54:39.693378 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:54:39.693378 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:54:39.693378 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:54:39.693378 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:54:39.693378 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:54:39.693378 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:54:39.693378 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 13 15:54:39.693378 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 13 15:54:39.693378 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Sep 13 15:54:39.693378 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Sep 13 15:54:39.693378 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 417/841] 10.222.0.50 () {56 vars in 1114 bytes} [Tue Sep 13 15:54:39 2022] GET /identity/v3/services/volumev3 => generated 89 bytes in 176 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 13 15:54:39.700268 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74937111-1e4f-43f8-89f1-3d13cb113446 None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:39.700772 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c791d832-d63f-455b-9e9d-3fcfcad01add None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:39.737378 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da338e57-a101-433b-ad26-29d0050bab3b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:39.738267 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da338e57-a101-433b-ad26-29d0050bab3b None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:39.738561 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da338e57-a101-433b-ad26-29d0050bab3b None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:39.738836 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da338e57-a101-433b-ad26-29d0050bab3b None admin] PATH_INFO: `/v3/users/swift` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:39.752811 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da338e57-a101-433b-ad26-29d0050bab3b None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:39.754376 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da338e57-a101-433b-ad26-29d0050bab3b None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:39.766960 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application [None req-da338e57-a101-433b-ad26-29d0050bab3b None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Sep 13 15:54:39.766960 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:54:39.766960 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:54:39.766960 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:54:39.766960 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:54:39.766960 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:54:39.766960 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:54:39.766960 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:54:39.766960 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:54:39.766960 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:54:39.766960 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:54:39.766960 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:54:39.766960 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 13 15:54:39.766960 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 13 15:54:39.766960 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 13 15:54:39.766960 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 13 15:54:39.767931 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:54:39.767931 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:54:39.767931 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 13 15:54:39.767931 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 13 15:54:39.767931 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 13 15:54:39.767931 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 13 15:54:39.767931 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:54:39.767931 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:54:39.767931 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:54:39.767931 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:54:39.767931 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:54:39.767931 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application with Lock( Sep 13 15:54:39.767931 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:54:39.767931 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:54:39.767931 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:54:39.767931 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:54:39.767931 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:54:39.767931 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:54:39.768860 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:54:39.768860 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:54:39.768860 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 13 15:54:39.768860 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 13 15:54:39.768860 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 13 15:54:39.768860 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 13 15:54:39.768860 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 13 15:54:39.768860 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 13 15:54:39.768860 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Sep 13 15:54:39.768860 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application Sep 13 15:54:39.771330 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 425/842] 10.222.0.50 () {56 vars in 1096 bytes} [Tue Sep 13 15:54:39 2022] GET /identity/v3/users/swift => generated 83 bytes in 202 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:54:39.777091 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da338e57-a101-433b-ad26-29d0050bab3b None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:39.817244 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:54:39.832331 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c791d832-d63f-455b-9e9d-3fcfcad01add None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:39.877537 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0f3beb7-71e6-438f-8c4d-c12382fd7a42 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:39.878160 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0f3beb7-71e6-438f-8c4d-c12382fd7a42 None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:39.878441 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0f3beb7-71e6-438f-8c4d-c12382fd7a42 None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:39.878719 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0f3beb7-71e6-438f-8c4d-c12382fd7a42 None admin] PATH_INFO: `/v3/services` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:39.879390 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0f3beb7-71e6-438f-8c4d-c12382fd7a42 None admin] RBAC: Adding query filter params (name=volumev3) {{(pid=81438) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:54:39.879705 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0f3beb7-71e6-438f-8c4d-c12382fd7a42 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:39.881059 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0f3beb7-71e6-438f-8c4d-c12382fd7a42 None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:39.894266 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 418/843] 10.222.0.50 () {56 vars in 1114 bytes} [Tue Sep 13 15:54:39 2022] GET /identity/v3/services?name=volumev3 => generated 125 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:54:39.896447 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0f3beb7-71e6-438f-8c4d-c12382fd7a42 None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:39.914149 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da338e57-a101-433b-ad26-29d0050bab3b None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:39.952944 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fbec068-c5a5-4425-adce-83d409aa4991 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:39.953467 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fbec068-c5a5-4425-adce-83d409aa4991 None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:39.953750 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fbec068-c5a5-4425-adce-83d409aa4991 None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:39.954023 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fbec068-c5a5-4425-adce-83d409aa4991 None admin] PATH_INFO: `/v3/users` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:39.954751 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fbec068-c5a5-4425-adce-83d409aa4991 None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=81437) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:54:39.955088 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fbec068-c5a5-4425-adce-83d409aa4991 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:39.956523 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fbec068-c5a5-4425-adce-83d409aa4991 None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:39.973051 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:54:39.977007 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 426/844] 10.222.0.50 () {56 vars in 1135 bytes} [Tue Sep 13 15:54:39 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 200 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 13 15:54:39.982275 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fbec068-c5a5-4425-adce-83d409aa4991 None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:40.036582 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0f3beb7-71e6-438f-8c4d-c12382fd7a42 None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:40.077849 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c69fc71-2004-4ecb-861a-39c0dde84464 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:40.079384 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c69fc71-2004-4ecb-861a-39c0dde84464 None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:40.080025 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c69fc71-2004-4ecb-861a-39c0dde84464 None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:40.080503 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c69fc71-2004-4ecb-861a-39c0dde84464 None admin] PATH_INFO: `/v3/services` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:40.081314 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c69fc71-2004-4ecb-861a-39c0dde84464 None admin] RBAC: Adding query filter params (type=volumev3) {{(pid=81438) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:54:40.081836 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c69fc71-2004-4ecb-861a-39c0dde84464 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:40.083626 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c69fc71-2004-4ecb-861a-39c0dde84464 None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:40.089422 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:54:40.095609 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 419/845] 10.222.0.50 () {56 vars in 1114 bytes} [Tue Sep 13 15:54:39 2022] GET /identity/v3/services?type=volumev3 => generated 125 bytes in 200 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 13 15:54:40.103347 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fbec068-c5a5-4425-adce-83d409aa4991 None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:40.143226 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c30670af-74d3-44c0-8165-62b3233fe0b4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:40.143845 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c30670af-74d3-44c0-8165-62b3233fe0b4 None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:40.144132 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c30670af-74d3-44c0-8165-62b3233fe0b4 None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:40.144459 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c30670af-74d3-44c0-8165-62b3233fe0b4 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:40.226350 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c30670af-74d3-44c0-8165-62b3233fe0b4 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:40.228516 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c30670af-74d3-44c0-8165-62b3233fe0b4 None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:40.332326 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:54:40.350575 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 427/846] 10.222.0.50 () {58 vars in 1282 bytes} [Tue Sep 13 15:54:39 2022] GET /identity/v3/auth/tokens => generated 1161 bytes in 368 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 13 15:54:40.357621 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c69fc71-2004-4ecb-861a-39c0dde84464 None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:40.470713 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c69fc71-2004-4ecb-861a-39c0dde84464 None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:40.489471 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:54:40.509801 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-433545ae-787c-4d5d-8e24-dba753290efa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:40.510796 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-433545ae-787c-4d5d-8e24-dba753290efa None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:40.511259 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-433545ae-787c-4d5d-8e24-dba753290efa None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:40.511675 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-433545ae-787c-4d5d-8e24-dba753290efa None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:40.523435 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-433545ae-787c-4d5d-8e24-dba753290efa None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:40.526003 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-433545ae-787c-4d5d-8e24-dba753290efa None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:40.541426 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application [None req-433545ae-787c-4d5d-8e24-dba753290efa None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:54:40.541426 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:54:40.541426 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:54:40.541426 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:54:40.541426 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:54:40.541426 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:54:40.541426 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:54:40.541426 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:54:40.541426 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:54:40.541426 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:54:40.541426 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:54:40.541426 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:54:40.541426 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 13 15:54:40.541426 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 13 15:54:40.541426 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 13 15:54:40.541426 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 13 15:54:40.542617 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:54:40.542617 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:54:40.542617 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:54:40.542617 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:54:40.542617 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:54:40.542617 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:54:40.542617 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:54:40.542617 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application with Lock( Sep 13 15:54:40.542617 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:54:40.542617 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:54:40.542617 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:54:40.542617 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:54:40.542617 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:54:40.542617 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:54:40.542617 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:54:40.542617 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:54:40.542617 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 13 15:54:40.543785 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 13 15:54:40.543785 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:54:40.543785 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Sep 13 15:54:40.545488 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 420/847] 10.222.0.50 () {56 vars in 1108 bytes} [Tue Sep 13 15:54:40 2022] GET /identity/v3/domains/Default => generated 87 bytes in 189 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:54:40.550923 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c30670af-74d3-44c0-8165-62b3233fe0b4 None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:40.660726 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c30670af-74d3-44c0-8165-62b3233fe0b4 None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:40.697874 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e7686df-00c6-458d-9748-0f45775837b3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:40.698736 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e7686df-00c6-458d-9748-0f45775837b3 None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:40.699184 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e7686df-00c6-458d-9748-0f45775837b3 None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:40.699601 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e7686df-00c6-458d-9748-0f45775837b3 None admin] PATH_INFO: `/v3/domains` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:40.700386 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e7686df-00c6-458d-9748-0f45775837b3 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81437) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:54:40.700863 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e7686df-00c6-458d-9748-0f45775837b3 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:40.702686 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e7686df-00c6-458d-9748-0f45775837b3 None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:40.717495 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 428/848] 10.222.0.50 () {56 vars in 1109 bytes} [Tue Sep 13 15:54:40 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:54:40.723290 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-433545ae-787c-4d5d-8e24-dba753290efa None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:40.840073 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-433545ae-787c-4d5d-8e24-dba753290efa None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:40.844135 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f819b488-130e-4dc6-a5eb-912bfe5d3ec1 None None] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:40.844428 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f819b488-130e-4dc6-a5eb-912bfe5d3ec1 None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:40.844711 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f819b488-130e-4dc6-a5eb-912bfe5d3ec1 None None] PATH_INFO: `/` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:40.847396 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 429/849] 10.222.0.50 () {54 vars in 927 bytes} [Tue Sep 13 15:54:40 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:54:40.851536 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7759d088-af22-4c74-ac93-1165212f8ee8 None None] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:40.851804 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7759d088-af22-4c74-ac93-1165212f8ee8 None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:40.852103 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7759d088-af22-4c74-ac93-1165212f8ee8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:40.873345 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.auth.core [None req-7759d088-af22-4c74-ac93-1165212f8ee8 None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81437) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:54:40.875500 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-7759d088-af22-4c74-ac93-1165212f8ee8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:54:40.897392 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad2677bb-70dd-4340-8e2a-cba47f0b8a28 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:40.899071 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad2677bb-70dd-4340-8e2a-cba47f0b8a28 None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:40.899674 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad2677bb-70dd-4340-8e2a-cba47f0b8a28 None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:40.900573 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad2677bb-70dd-4340-8e2a-cba47f0b8a28 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:40.911073 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad2677bb-70dd-4340-8e2a-cba47f0b8a28 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:40.913444 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad2677bb-70dd-4340-8e2a-cba47f0b8a28 None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:40.926466 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application [None req-ad2677bb-70dd-4340-8e2a-cba47f0b8a28 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 13 15:54:40.926466 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:54:40.926466 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:54:40.926466 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:54:40.926466 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:54:40.926466 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:54:40.926466 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:54:40.926466 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:54:40.926466 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:54:40.926466 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:54:40.926466 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:54:40.926466 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:54:40.926466 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 13 15:54:40.926466 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 13 15:54:40.926466 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 13 15:54:40.927476 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 13 15:54:40.927476 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:54:40.927476 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:54:40.927476 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:54:40.927476 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:54:40.927476 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:54:40.927476 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:54:40.927476 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:54:40.927476 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application with Lock( Sep 13 15:54:40.927476 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:54:40.927476 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:54:40.927476 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:54:40.927476 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:54:40.927476 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:54:40.927476 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:54:40.927476 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:54:40.927476 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:54:40.928395 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 13 15:54:40.928395 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 13 15:54:40.928395 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 13 15:54:40.928395 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 13 15:54:40.928395 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 13 15:54:40.928395 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 13 15:54:40.928395 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 13 15:54:40.928395 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Sep 13 15:54:40.928395 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 421/850] 10.222.0.50 () {56 vars in 1111 bytes} [Tue Sep 13 15:54:40 2022] GET /identity/v3/projects/service => generated 88 bytes in 205 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:54:40.934517 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-386eb7cf-f98c-4ef7-a208-105473690771 None None] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:40.934797 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-386eb7cf-f98c-4ef7-a208-105473690771 None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:40.935117 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-386eb7cf-f98c-4ef7-a208-105473690771 None None] PATH_INFO: `/` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:40.936402 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 422/851] 10.222.0.50 () {54 vars in 927 bytes} [Tue Sep 13 15:54:40 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 13 15:54:40.937092 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-386eb7cf-f98c-4ef7-a208-105473690771 None None] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:40.937828 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:54:40.969857 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 430/852] 10.222.0.50 () {58 vars in 1024 bytes} [Tue Sep 13 15:54:40 2022] POST /identity/v3/auth/tokens => generated 1161 bytes in 119 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 13 15:54:40.970055 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:54:40.971926 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-177ad487-938d-4dff-b177-6a8dbb6bf76c None None] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:40.972456 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-177ad487-938d-4dff-b177-6a8dbb6bf76c None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:40.972947 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-177ad487-938d-4dff-b177-6a8dbb6bf76c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:40.995730 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.auth.core [None req-177ad487-938d-4dff-b177-6a8dbb6bf76c None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81437) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:54:40.997650 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-177ad487-938d-4dff-b177-6a8dbb6bf76c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:54:41.047567 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-386eb7cf-f98c-4ef7-a208-105473690771 None None] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:41.085770 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 431/853] 10.222.0.50 () {58 vars in 1024 bytes} [Tue Sep 13 15:54:40 2022] POST /identity/v3/auth/tokens => generated 1161 bytes in 115 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 13 15:54:41.085986 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f2af04c-5601-4ba3-a51b-302c4dc147cb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:41.086765 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f2af04c-5601-4ba3-a51b-302c4dc147cb None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:41.086925 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f2af04c-5601-4ba3-a51b-302c4dc147cb None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:41.087220 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f2af04c-5601-4ba3-a51b-302c4dc147cb None admin] PATH_INFO: `/v3/projects` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:41.087911 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f2af04c-5601-4ba3-a51b-302c4dc147cb None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81438) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:54:41.088215 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f2af04c-5601-4ba3-a51b-302c4dc147cb None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:41.089632 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f2af04c-5601-4ba3-a51b-302c4dc147cb None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:41.110822 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 423/854] 10.222.0.50 () {56 vars in 1148 bytes} [Tue Sep 13 15:54:40 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:54:41.116600 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-177ad487-938d-4dff-b177-6a8dbb6bf76c None None] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:41.219514 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-177ad487-938d-4dff-b177-6a8dbb6bf76c None None] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:41.229467 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:54:41.252886 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f2af04c-5601-4ba3-a51b-302c4dc147cb None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:41.255466 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e17aab98-4b18-41a8-9dcc-aa716dd760ae None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:41.256100 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e17aab98-4b18-41a8-9dcc-aa716dd760ae None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:41.256522 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e17aab98-4b18-41a8-9dcc-aa716dd760ae None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:41.256805 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e17aab98-4b18-41a8-9dcc-aa716dd760ae None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:41.257533 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e17aab98-4b18-41a8-9dcc-aa716dd760ae None admin] RBAC: Adding query filter params (role.id=031d4e1a23e34f71858fc4a6dfbd4ccb, scope.project.id=5811e3b83a1e462e9320cad79a3ad791, user.id=b00ab0d6321d483188bc69deaddcdbbc) {{(pid=81437) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:54:41.258795 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e17aab98-4b18-41a8-9dcc-aa716dd760ae None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:41.260264 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e17aab98-4b18-41a8-9dcc-aa716dd760ae None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:41.267648 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 432/855] 10.222.0.50 () {56 vars in 1374 bytes} [Tue Sep 13 15:54:41 2022] GET /identity/v3/role_assignments?user.id=b00ab0d6321d483188bc69deaddcdbbc&scope.project.id=5811e3b83a1e462e9320cad79a3ad791&role.id=031d4e1a23e34f71858fc4a6dfbd4ccb => generated 608 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:54:41.363968 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:54:41.373504 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e17aab98-4b18-41a8-9dcc-aa716dd760ae None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:41.374448 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f2af04c-5601-4ba3-a51b-302c4dc147cb None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:41.420684 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e19eec9a-ab15-4088-a33b-a6837c26b530 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:41.421968 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e19eec9a-ab15-4088-a33b-a6837c26b530 None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:41.422261 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e19eec9a-ab15-4088-a33b-a6837c26b530 None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:41.422866 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e19eec9a-ab15-4088-a33b-a6837c26b530 None admin] PATH_INFO: `/v3/services/compute_legacy` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:41.423778 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e19eec9a-ab15-4088-a33b-a6837c26b530 None admin] RBAC: Authorizing `identity:get_service(service_id=compute_legacy)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:41.425213 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e19eec9a-ab15-4088-a33b-a6837c26b530 None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:41.440164 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application [None req-e19eec9a-ab15-4088-a33b-a6837c26b530 None admin] Could not find service: compute_legacy.: keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Sep 13 15:54:41.440164 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:54:41.440164 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 13 15:54:41.440164 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 13 15:54:41.440164 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 13 15:54:41.440164 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 13 15:54:41.440164 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 13 15:54:41.440164 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 13 15:54:41.440164 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Sep 13 15:54:41.440164 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Sep 13 15:54:41.440164 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 13 15:54:41.440164 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Sep 13 15:54:41.440164 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:54:41.440164 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:54:41.440164 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:54:41.440164 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:54:41.441172 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:54:41.441172 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:54:41.441172 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:54:41.441172 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:54:41.441172 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:54:41.441172 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:54:41.441172 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:54:41.441172 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 13 15:54:41.441172 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 13 15:54:41.441172 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 13 15:54:41.441172 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 13 15:54:41.441172 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:54:41.441172 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:54:41.441172 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:54:41.441172 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:54:41.441172 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:54:41.441172 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:54:41.442065 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:54:41.442065 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application with Lock( Sep 13 15:54:41.442065 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:54:41.442065 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:54:41.442065 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:54:41.442065 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:54:41.442065 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:54:41.442065 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:54:41.442065 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:54:41.442065 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:54:41.442065 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 13 15:54:41.442065 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 13 15:54:41.442065 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Sep 13 15:54:41.442065 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Sep 13 15:54:41.447120 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 424/856] 10.222.0.50 () {56 vars in 1132 bytes} [Tue Sep 13 15:54:41 2022] GET /identity/v3/services/compute_legacy => generated 95 bytes in 189 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:54:41.449562 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e19eec9a-ab15-4088-a33b-a6837c26b530 None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:41.505013 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e17aab98-4b18-41a8-9dcc-aa716dd760ae None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:41.547508 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0045d272-7d37-4f76-8dec-1e85d71cc41b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:41.548156 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0045d272-7d37-4f76-8dec-1e85d71cc41b None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:41.548433 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0045d272-7d37-4f76-8dec-1e85d71cc41b None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:41.548710 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0045d272-7d37-4f76-8dec-1e85d71cc41b None admin] PATH_INFO: `/v3/roles/service` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:41.560803 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0045d272-7d37-4f76-8dec-1e85d71cc41b None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 13 15:54:41.566543 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application [None req-0045d272-7d37-4f76-8dec-1e85d71cc41b None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 13 15:54:41.566543 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:54:41.566543 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:54:41.566543 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:54:41.566543 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:54:41.566543 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:54:41.566543 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:54:41.566543 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:54:41.566543 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:54:41.566543 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:54:41.566543 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:54:41.566543 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:54:41.566543 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 13 15:54:41.566543 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 13 15:54:41.566543 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 13 15:54:41.566543 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 13 15:54:41.567518 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 13 15:54:41.567518 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 13 15:54:41.567518 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 13 15:54:41.567518 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 13 15:54:41.567518 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:54:41.567518 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:54:41.567518 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:54:41.567518 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:54:41.567518 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:54:41.567518 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:54:41.567518 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:54:41.567518 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application with Lock( Sep 13 15:54:41.567518 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:54:41.567518 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:54:41.567518 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:54:41.567518 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:54:41.568648 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:54:41.568648 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:54:41.568648 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:54:41.568648 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:54:41.568648 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 13 15:54:41.568648 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 13 15:54:41.568648 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 13 15:54:41.568648 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 13 15:54:41.568648 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 13 15:54:41.568648 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 13 15:54:41.568648 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 13 15:54:41.568648 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application Sep 13 15:54:41.568648 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 433/857] 10.222.0.50 () {56 vars in 1102 bytes} [Tue Sep 13 15:54:41 2022] GET /identity/v3/roles/service => generated 85 bytes in 195 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:54:41.574326 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0045d272-7d37-4f76-8dec-1e85d71cc41b None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:41.592302 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e19eec9a-ab15-4088-a33b-a6837c26b530 None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:41.608716 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:54:41.636725 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49eece0b-111e-434f-a18b-5d756c024dca None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:41.637404 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49eece0b-111e-434f-a18b-5d756c024dca None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:41.637786 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49eece0b-111e-434f-a18b-5d756c024dca None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:41.638064 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49eece0b-111e-434f-a18b-5d756c024dca None admin] PATH_INFO: `/v3/services` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:41.638695 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49eece0b-111e-434f-a18b-5d756c024dca None admin] RBAC: Adding query filter params (name=compute_legacy) {{(pid=81438) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:54:41.639027 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49eece0b-111e-434f-a18b-5d756c024dca None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:41.641064 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49eece0b-111e-434f-a18b-5d756c024dca None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:41.646706 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 425/858] 10.222.0.50 () {56 vars in 1126 bytes} [Tue Sep 13 15:54:41 2022] GET /identity/v3/services?name=compute_legacy => generated 131 bytes in 197 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:54:41.652306 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49eece0b-111e-434f-a18b-5d756c024dca None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:41.672033 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:54:41.691758 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0045d272-7d37-4f76-8dec-1e85d71cc41b None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:41.733866 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bbec141-69ba-4f3d-b6fa-7c233a03bdfb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:41.735261 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bbec141-69ba-4f3d-b6fa-7c233a03bdfb None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:41.735871 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bbec141-69ba-4f3d-b6fa-7c233a03bdfb None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:41.736457 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bbec141-69ba-4f3d-b6fa-7c233a03bdfb None admin] PATH_INFO: `/v3/roles` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:41.737304 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2bbec141-69ba-4f3d-b6fa-7c233a03bdfb None admin] RBAC: Adding query filter params (name=service) {{(pid=81437) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:54:41.738057 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2bbec141-69ba-4f3d-b6fa-7c233a03bdfb None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:41.740422 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2bbec141-69ba-4f3d-b6fa-7c233a03bdfb None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:41.752621 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 434/859] 10.222.0.50 () {56 vars in 1103 bytes} [Tue Sep 13 15:54:41 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 13 15:54:41.767008 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bbec141-69ba-4f3d-b6fa-7c233a03bdfb None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:41.767170 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49eece0b-111e-434f-a18b-5d756c024dca None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:41.798467 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df644a9c-bd04-4fcd-a079-be9be35662fc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:41.799286 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df644a9c-bd04-4fcd-a079-be9be35662fc None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:41.799697 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df644a9c-bd04-4fcd-a079-be9be35662fc None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:41.800075 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df644a9c-bd04-4fcd-a079-be9be35662fc None admin] PATH_INFO: `/v3/services` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:41.800792 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df644a9c-bd04-4fcd-a079-be9be35662fc None admin] RBAC: Adding query filter params (type=compute_legacy) {{(pid=81438) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:54:41.801194 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df644a9c-bd04-4fcd-a079-be9be35662fc None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:41.802627 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df644a9c-bd04-4fcd-a079-be9be35662fc None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:41.815228 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 426/860] 10.222.0.50 () {56 vars in 1126 bytes} [Tue Sep 13 15:54:41 2022] GET /identity/v3/services?type=compute_legacy => generated 131 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 13 15:54:41.873281 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:54:41.881508 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bbec141-69ba-4f3d-b6fa-7c233a03bdfb None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:41.920621 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f41b2f5-5541-4435-a5b7-6138628bf90b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:41.921625 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f41b2f5-5541-4435-a5b7-6138628bf90b None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:41.921907 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f41b2f5-5541-4435-a5b7-6138628bf90b None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:41.922180 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f41b2f5-5541-4435-a5b7-6138628bf90b None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:41.933813 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f41b2f5-5541-4435-a5b7-6138628bf90b None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:41.935308 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f41b2f5-5541-4435-a5b7-6138628bf90b None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:41.952249 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application [None req-6f41b2f5-5541-4435-a5b7-6138628bf90b None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:54:41.952249 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:54:41.952249 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:54:41.952249 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:54:41.952249 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:54:41.952249 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:54:41.952249 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:54:41.952249 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:54:41.952249 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:54:41.952249 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:54:41.952249 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:54:41.952249 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:54:41.952249 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 13 15:54:41.952249 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 13 15:54:41.952249 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 13 15:54:41.952249 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 13 15:54:41.953287 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:54:41.953287 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:54:41.953287 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:54:41.953287 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:54:41.953287 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:54:41.953287 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:54:41.953287 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:54:41.953287 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application with Lock( Sep 13 15:54:41.953287 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:54:41.953287 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:54:41.953287 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:54:41.953287 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:54:41.953287 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:54:41.953287 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:54:41.953287 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:54:41.953287 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:54:41.953287 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 13 15:54:41.954183 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 13 15:54:41.954183 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:54:41.954183 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application Sep 13 15:54:41.954183 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 435/861] 10.222.0.50 () {56 vars in 1108 bytes} [Tue Sep 13 15:54:41 2022] GET /identity/v3/domains/Default => generated 87 bytes in 196 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 13 15:54:41.959855 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df644a9c-bd04-4fcd-a079-be9be35662fc None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:42.059514 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:54:42.081471 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df644a9c-bd04-4fcd-a079-be9be35662fc None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:42.116596 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-691ef965-6815-47f8-b4b5-8ee11378510c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:42.117430 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-691ef965-6815-47f8-b4b5-8ee11378510c None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:42.118097 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-691ef965-6815-47f8-b4b5-8ee11378510c None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:42.118344 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-691ef965-6815-47f8-b4b5-8ee11378510c None admin] PATH_INFO: `/v3/domains` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:42.119004 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-691ef965-6815-47f8-b4b5-8ee11378510c None admin] RBAC: Adding query filter params (name=Default) {{(pid=81438) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:54:42.119304 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-691ef965-6815-47f8-b4b5-8ee11378510c None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:42.120665 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-691ef965-6815-47f8-b4b5-8ee11378510c None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:42.135871 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 427/862] 10.222.0.50 () {56 vars in 1109 bytes} [Tue Sep 13 15:54:41 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:54:42.141367 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f41b2f5-5541-4435-a5b7-6138628bf90b None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:42.246852 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f41b2f5-5541-4435-a5b7-6138628bf90b None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:42.284349 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fb0aedb-b6b5-4d8a-a1ec-7ba16b9d86eb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:42.284990 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fb0aedb-b6b5-4d8a-a1ec-7ba16b9d86eb None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:42.285269 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fb0aedb-b6b5-4d8a-a1ec-7ba16b9d86eb None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:42.285545 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fb0aedb-b6b5-4d8a-a1ec-7ba16b9d86eb None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:42.294379 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fb0aedb-b6b5-4d8a-a1ec-7ba16b9d86eb None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:42.295867 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fb0aedb-b6b5-4d8a-a1ec-7ba16b9d86eb None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:42.308322 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application [None req-1fb0aedb-b6b5-4d8a-a1ec-7ba16b9d86eb None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Sep 13 15:54:42.308322 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:54:42.308322 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:54:42.308322 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:54:42.308322 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:54:42.308322 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:54:42.308322 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:54:42.308322 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:54:42.308322 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:54:42.308322 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:54:42.308322 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:54:42.308322 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:54:42.308322 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 13 15:54:42.308322 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 13 15:54:42.308322 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 13 15:54:42.308322 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 13 15:54:42.310568 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:54:42.310568 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:54:42.310568 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 13 15:54:42.310568 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 13 15:54:42.310568 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 13 15:54:42.310568 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 13 15:54:42.310568 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:54:42.310568 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:54:42.310568 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:54:42.310568 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:54:42.310568 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:54:42.310568 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application with Lock( Sep 13 15:54:42.310568 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:54:42.310568 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:54:42.310568 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:54:42.310568 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:54:42.310568 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:54:42.310568 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:54:42.312895 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:54:42.312895 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:54:42.312895 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 13 15:54:42.312895 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 13 15:54:42.312895 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 13 15:54:42.312895 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 13 15:54:42.312895 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 13 15:54:42.312895 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 13 15:54:42.312895 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Sep 13 15:54:42.312895 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application Sep 13 15:54:42.312895 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 436/863] 10.222.0.50 () {56 vars in 1117 bytes} [Tue Sep 13 15:54:42 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 169 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:54:42.318946 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-691ef965-6815-47f8-b4b5-8ee11378510c None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:42.435498 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-691ef965-6815-47f8-b4b5-8ee11378510c None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:42.471964 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-454ce6fd-f184-4459-b4d9-9dd0a00fde3b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:42.472572 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-454ce6fd-f184-4459-b4d9-9dd0a00fde3b None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:42.472851 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-454ce6fd-f184-4459-b4d9-9dd0a00fde3b None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:42.473123 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-454ce6fd-f184-4459-b4d9-9dd0a00fde3b None admin] PATH_INFO: `/v3/users` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:42.473840 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-454ce6fd-f184-4459-b4d9-9dd0a00fde3b None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=81438) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:54:42.474145 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-454ce6fd-f184-4459-b4d9-9dd0a00fde3b None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:42.475594 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-454ce6fd-f184-4459-b4d9-9dd0a00fde3b None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:42.490006 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:54:42.495087 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 428/864] 10.222.0.50 () {56 vars in 1149 bytes} [Tue Sep 13 15:54:42 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 180 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:54:42.500650 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fb0aedb-b6b5-4d8a-a1ec-7ba16b9d86eb None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:42.526131 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb350b99-e7a2-49e4-af56-ad1b74791db1 None None] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:42.526440 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb350b99-e7a2-49e4-af56-ad1b74791db1 None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:42.526684 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb350b99-e7a2-49e4-af56-ad1b74791db1 None None] PATH_INFO: `/` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:42.528298 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 429/865] 10.222.0.50 () {54 vars in 927 bytes} [Tue Sep 13 15:54:42 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 13 15:54:42.531280 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:54:42.533266 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97614f88-5009-420b-8afe-4dccb7b3649e None None] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:42.533551 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97614f88-5009-420b-8afe-4dccb7b3649e None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:42.533827 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97614f88-5009-420b-8afe-4dccb7b3649e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:42.552423 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.auth.core [None req-97614f88-5009-420b-8afe-4dccb7b3649e None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81438) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:54:42.553959 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-97614f88-5009-420b-8afe-4dccb7b3649e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:54:42.612511 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fb0aedb-b6b5-4d8a-a1ec-7ba16b9d86eb None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:42.644111 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 430/866] 10.222.0.50 () {58 vars in 1024 bytes} [Tue Sep 13 15:54:42 2022] POST /identity/v3/auth/tokens => generated 1161 bytes in 111 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 13 15:54:42.654342 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0cc0d59-b9fc-4a75-a686-4f16b8505078 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:42.655245 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0cc0d59-b9fc-4a75-a686-4f16b8505078 None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:42.655765 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0cc0d59-b9fc-4a75-a686-4f16b8505078 None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:42.656264 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0cc0d59-b9fc-4a75-a686-4f16b8505078 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:42.727559 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0cc0d59-b9fc-4a75-a686-4f16b8505078 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:42.729356 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0cc0d59-b9fc-4a75-a686-4f16b8505078 None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:42.737773 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:54:42.834350 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 437/867] 10.222.0.50 () {58 vars in 1282 bytes} [Tue Sep 13 15:54:42 2022] GET /identity/v3/auth/tokens => generated 1161 bytes in 334 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 13 15:54:42.840389 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97614f88-5009-420b-8afe-4dccb7b3649e None None] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:42.922704 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0cc0d59-b9fc-4a75-a686-4f16b8505078 None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:42.946012 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97614f88-5009-420b-8afe-4dccb7b3649e None None] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:42.980421 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:54:42.987910 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cd52db0-4f05-4867-a97e-b9ff842b1c30 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:42.988868 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cd52db0-4f05-4867-a97e-b9ff842b1c30 None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:42.989417 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cd52db0-4f05-4867-a97e-b9ff842b1c30 None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:42.989958 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cd52db0-4f05-4867-a97e-b9ff842b1c30 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:43.001094 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2cd52db0-4f05-4867-a97e-b9ff842b1c30 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:43.002707 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2cd52db0-4f05-4867-a97e-b9ff842b1c30 None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:43.019627 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application [None req-2cd52db0-4f05-4867-a97e-b9ff842b1c30 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:54:43.019627 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:54:43.019627 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:54:43.019627 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:54:43.019627 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:54:43.019627 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:54:43.019627 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:54:43.019627 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:54:43.019627 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:54:43.019627 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:54:43.019627 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:54:43.019627 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:54:43.019627 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 13 15:54:43.019627 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 13 15:54:43.019627 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 13 15:54:43.019627 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 13 15:54:43.020611 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:54:43.020611 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:54:43.020611 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:54:43.020611 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:54:43.020611 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:54:43.020611 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:54:43.020611 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:54:43.020611 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application with Lock( Sep 13 15:54:43.020611 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:54:43.020611 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:54:43.020611 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:54:43.020611 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:54:43.020611 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:54:43.020611 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:54:43.020611 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:54:43.020611 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:54:43.020611 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 13 15:54:43.021517 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 13 15:54:43.021517 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:54:43.021517 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Sep 13 15:54:43.024198 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 431/868] 10.222.0.50 () {56 vars in 1108 bytes} [Tue Sep 13 15:54:42 2022] GET /identity/v3/domains/Default => generated 87 bytes in 184 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:54:43.029919 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cd52db0-4f05-4867-a97e-b9ff842b1c30 None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:43.043730 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0cc0d59-b9fc-4a75-a686-4f16b8505078 None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:43.052833 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:54:43.085870 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1402b92-a8d5-4924-b815-270cb50e6037 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:43.087700 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1402b92-a8d5-4924-b815-270cb50e6037 None admin] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:43.087833 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1402b92-a8d5-4924-b815-270cb50e6037 None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:43.088073 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1402b92-a8d5-4924-b815-270cb50e6037 None admin] PATH_INFO: `/v3/services` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:43.088747 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1402b92-a8d5-4924-b815-270cb50e6037 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:43.090244 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1402b92-a8d5-4924-b815-270cb50e6037 None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:43.115112 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 438/869] 10.222.0.50 () {60 vars in 1141 bytes} [Tue Sep 13 15:54:42 2022] POST /identity/v3/services => generated 251 bytes in 186 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 13 15:54:43.156435 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cd52db0-4f05-4867-a97e-b9ff842b1c30 None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:43.204399 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed49f51e-7323-4141-b016-84636f1fed40 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:43.205427 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed49f51e-7323-4141-b016-84636f1fed40 None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:43.205920 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed49f51e-7323-4141-b016-84636f1fed40 None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:43.206402 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed49f51e-7323-4141-b016-84636f1fed40 None admin] PATH_INFO: `/v3/domains` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:43.207294 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed49f51e-7323-4141-b016-84636f1fed40 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81438) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:54:43.207836 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed49f51e-7323-4141-b016-84636f1fed40 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:43.209519 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed49f51e-7323-4141-b016-84636f1fed40 None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:43.224071 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 432/870] 10.222.0.50 () {56 vars in 1109 bytes} [Tue Sep 13 15:54:43 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 194 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:54:43.230293 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1402b92-a8d5-4924-b815-270cb50e6037 None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:43.343335 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1402b92-a8d5-4924-b815-270cb50e6037 None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:43.380132 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4962bafe-7ae8-443a-b47e-c65426058779 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:43.381158 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4962bafe-7ae8-443a-b47e-c65426058779 None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:43.381648 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4962bafe-7ae8-443a-b47e-c65426058779 None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:43.382130 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4962bafe-7ae8-443a-b47e-c65426058779 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:43.389418 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4962bafe-7ae8-443a-b47e-c65426058779 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:43.391166 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4962bafe-7ae8-443a-b47e-c65426058779 None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:43.402856 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application [None req-4962bafe-7ae8-443a-b47e-c65426058779 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 13 15:54:43.402856 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:54:43.402856 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:54:43.402856 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:54:43.402856 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:54:43.402856 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:54:43.402856 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:54:43.402856 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:54:43.402856 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:54:43.402856 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:54:43.402856 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:54:43.402856 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:54:43.402856 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 13 15:54:43.402856 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 13 15:54:43.402856 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 13 15:54:43.403840 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 13 15:54:43.403840 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:54:43.403840 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:54:43.403840 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:54:43.403840 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:54:43.403840 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:54:43.403840 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:54:43.403840 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:54:43.403840 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application with Lock( Sep 13 15:54:43.403840 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:54:43.403840 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:54:43.403840 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:54:43.403840 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:54:43.403840 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:54:43.403840 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:54:43.403840 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:54:43.403840 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:54:43.404809 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 13 15:54:43.404809 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 13 15:54:43.404809 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 13 15:54:43.404809 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 13 15:54:43.404809 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 13 15:54:43.404809 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 13 15:54:43.404809 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 13 15:54:43.404809 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application Sep 13 15:54:43.407225 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 439/871] 10.222.0.50 () {56 vars in 1111 bytes} [Tue Sep 13 15:54:43 2022] GET /identity/v3/projects/service => generated 88 bytes in 177 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:54:43.413258 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed49f51e-7323-4141-b016-84636f1fed40 None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:43.457223 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:54:43.522718 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed49f51e-7323-4141-b016-84636f1fed40 None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:43.563548 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe691138-3cba-4adc-9097-97e0f319a3b4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:43.564451 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe691138-3cba-4adc-9097-97e0f319a3b4 None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:43.564875 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe691138-3cba-4adc-9097-97e0f319a3b4 None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:43.565302 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe691138-3cba-4adc-9097-97e0f319a3b4 None admin] PATH_INFO: `/v3/projects` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:43.566209 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe691138-3cba-4adc-9097-97e0f319a3b4 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81438) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:54:43.566666 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe691138-3cba-4adc-9097-97e0f319a3b4 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:43.568266 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe691138-3cba-4adc-9097-97e0f319a3b4 None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:43.581200 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 433/872] 10.222.0.50 () {56 vars in 1148 bytes} [Tue Sep 13 15:54:43 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:54:43.587434 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4962bafe-7ae8-443a-b47e-c65426058779 None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:43.617054 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:54:43.637736 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84bfdf90-17d8-44ed-89e3-cc461942111d None None] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:43.638251 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84bfdf90-17d8-44ed-89e3-cc461942111d None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:43.638708 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84bfdf90-17d8-44ed-89e3-cc461942111d None None] PATH_INFO: `/` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:43.640448 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 434/873] 10.222.0.50 () {54 vars in 927 bytes} [Tue Sep 13 15:54:43 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:54:43.645457 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5d03301-6dba-4054-bc3c-a6446d72300b None None] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:43.645929 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5d03301-6dba-4054-bc3c-a6446d72300b None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:43.646382 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5d03301-6dba-4054-bc3c-a6446d72300b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:43.667373 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.auth.core [None req-c5d03301-6dba-4054-bc3c-a6446d72300b None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81438) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:54:43.669129 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-c5d03301-6dba-4054-bc3c-a6446d72300b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:54:43.704183 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4962bafe-7ae8-443a-b47e-c65426058779 None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:43.743529 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee7dd54b-0a69-45a9-a10c-088fffafd571 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:43.743972 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee7dd54b-0a69-45a9-a10c-088fffafd571 None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:43.744258 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee7dd54b-0a69-45a9-a10c-088fffafd571 None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:43.744537 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee7dd54b-0a69-45a9-a10c-088fffafd571 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:43.745317 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee7dd54b-0a69-45a9-a10c-088fffafd571 None admin] RBAC: Adding query filter params (role.id=da847124d9374ac88b27968451c486ab, scope.project.id=5811e3b83a1e462e9320cad79a3ad791, user.id=f8364b1a190d4218bf353a89bdb81448) {{(pid=81437) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:54:43.745637 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee7dd54b-0a69-45a9-a10c-088fffafd571 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:43.746962 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 435/874] 10.222.0.50 () {58 vars in 1024 bytes} [Tue Sep 13 15:54:43 2022] POST /identity/v3/auth/tokens => generated 1262 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 13 15:54:43.747131 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee7dd54b-0a69-45a9-a10c-088fffafd571 None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:43.762933 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 440/875] 10.222.0.50 () {56 vars in 1374 bytes} [Tue Sep 13 15:54:43 2022] GET /identity/v3/role_assignments?user.id=f8364b1a190d4218bf353a89bdb81448&scope.project.id=5811e3b83a1e462e9320cad79a3ad791&role.id=da847124d9374ac88b27968451c486ab => generated 608 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:54:44.022220 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5d03301-6dba-4054-bc3c-a6446d72300b None None] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:44.052788 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:54:44.114218 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff341527-c4a1-4244-8e95-b0ea2db71b68 None None] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:44.114557 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff341527-c4a1-4244-8e95-b0ea2db71b68 None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:44.114763 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff341527-c4a1-4244-8e95-b0ea2db71b68 None None] PATH_INFO: `/` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:44.116502 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 441/876] 10.222.0.50 () {54 vars in 927 bytes} [Tue Sep 13 15:54:44 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:54:44.123411 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adb6e170-1805-43b3-8bf1-c508b6e4ddf7 None None] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:44.123411 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adb6e170-1805-43b3-8bf1-c508b6e4ddf7 None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:44.123931 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adb6e170-1805-43b3-8bf1-c508b6e4ddf7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:44.138021 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5d03301-6dba-4054-bc3c-a6446d72300b None None] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:44.147666 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.auth.core [None req-adb6e170-1805-43b3-8bf1-c508b6e4ddf7 None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81437) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:54:44.150292 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-adb6e170-1805-43b3-8bf1-c508b6e4ddf7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:54:44.182767 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-530cdf86-59d8-4cf5-b858-1d5e60fc5563 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:44.183461 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-530cdf86-59d8-4cf5-b858-1d5e60fc5563 None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:44.183745 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-530cdf86-59d8-4cf5-b858-1d5e60fc5563 None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:44.184018 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-530cdf86-59d8-4cf5-b858-1d5e60fc5563 None admin] PATH_INFO: `/v3/services/object-store` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:44.184611 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-530cdf86-59d8-4cf5-b858-1d5e60fc5563 None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:44.186496 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-530cdf86-59d8-4cf5-b858-1d5e60fc5563 None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:44.199922 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application [None req-530cdf86-59d8-4cf5-b858-1d5e60fc5563 None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Sep 13 15:54:44.199922 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:54:44.199922 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 13 15:54:44.199922 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 13 15:54:44.199922 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 13 15:54:44.199922 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 13 15:54:44.199922 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 13 15:54:44.199922 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 13 15:54:44.199922 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Sep 13 15:54:44.199922 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Sep 13 15:54:44.199922 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 13 15:54:44.199922 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Sep 13 15:54:44.199922 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:54:44.199922 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:54:44.199922 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:54:44.199922 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:54:44.199922 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:54:44.201175 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:54:44.201175 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:54:44.201175 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:54:44.201175 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:54:44.201175 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:54:44.201175 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:54:44.201175 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 13 15:54:44.201175 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 13 15:54:44.201175 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 13 15:54:44.201175 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 13 15:54:44.201175 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:54:44.201175 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:54:44.201175 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:54:44.201175 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:54:44.201175 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:54:44.201175 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:54:44.202059 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:54:44.202059 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application with Lock( Sep 13 15:54:44.202059 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:54:44.202059 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:54:44.202059 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:54:44.202059 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:54:44.202059 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:54:44.202059 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:54:44.202059 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:54:44.202059 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:54:44.202059 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 13 15:54:44.202059 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 13 15:54:44.202059 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Sep 13 15:54:44.202059 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Sep 13 15:54:44.202059 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 436/877] 10.222.0.50 () {56 vars in 1126 bytes} [Tue Sep 13 15:54:44 2022] GET /identity/v3/services/object-store => generated 93 bytes in 178 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 13 15:54:44.205404 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-530cdf86-59d8-4cf5-b858-1d5e60fc5563 None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:44.250578 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 442/878] 10.222.0.50 () {58 vars in 1024 bytes} [Tue Sep 13 15:54:44 2022] POST /identity/v3/auth/tokens => generated 1262 bytes in 128 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 13 15:54:44.312712 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:54:44.324509 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-530cdf86-59d8-4cf5-b858-1d5e60fc5563 None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:44.359753 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bdca636-411d-4619-b45c-b1ce33090637 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:44.360367 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bdca636-411d-4619-b45c-b1ce33090637 None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:44.360647 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bdca636-411d-4619-b45c-b1ce33090637 None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:44.360919 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bdca636-411d-4619-b45c-b1ce33090637 None admin] PATH_INFO: `/v3/services` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:44.361547 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bdca636-411d-4619-b45c-b1ce33090637 None admin] RBAC: Adding query filter params (name=object-store) {{(pid=81438) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:54:44.361851 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bdca636-411d-4619-b45c-b1ce33090637 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:44.363314 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bdca636-411d-4619-b45c-b1ce33090637 None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:44.369108 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 437/879] 10.222.0.50 () {56 vars in 1122 bytes} [Tue Sep 13 15:54:44 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 13 15:54:44.374839 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adb6e170-1805-43b3-8bf1-c508b6e4ddf7 None None] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:44.397141 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:54:44.479934 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adb6e170-1805-43b3-8bf1-c508b6e4ddf7 None None] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:44.515876 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5edc6c6b-365e-4e31-a413-79e7d83a40ad None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:44.516486 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5edc6c6b-365e-4e31-a413-79e7d83a40ad None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:44.516764 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5edc6c6b-365e-4e31-a413-79e7d83a40ad None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:44.517041 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5edc6c6b-365e-4e31-a413-79e7d83a40ad None admin] PATH_INFO: `/v3/services` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:44.517673 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5edc6c6b-365e-4e31-a413-79e7d83a40ad None admin] RBAC: Adding query filter params (type=object-store) {{(pid=81437) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:54:44.517978 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5edc6c6b-365e-4e31-a413-79e7d83a40ad None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:44.519403 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5edc6c6b-365e-4e31-a413-79e7d83a40ad None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:44.525452 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 443/880] 10.222.0.50 () {56 vars in 1122 bytes} [Tue Sep 13 15:54:44 2022] GET /identity/v3/services?type=object-store => generated 129 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:54:44.530033 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bdca636-411d-4619-b45c-b1ce33090637 None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:44.645805 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bdca636-411d-4619-b45c-b1ce33090637 None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:44.683135 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9a797b6-a4e4-4436-bf28-67bc95697f74 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:44.683812 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9a797b6-a4e4-4436-bf28-67bc95697f74 None admin] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:44.684094 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9a797b6-a4e4-4436-bf28-67bc95697f74 None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:44.684363 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9a797b6-a4e4-4436-bf28-67bc95697f74 None admin] PATH_INFO: `/v3/services` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:44.685029 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9a797b6-a4e4-4436-bf28-67bc95697f74 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:44.686482 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9a797b6-a4e4-4436-bf28-67bc95697f74 None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:44.706314 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 438/881] 10.222.0.50 () {60 vars in 1141 bytes} [Tue Sep 13 15:54:44 2022] POST /identity/v3/services => generated 269 bytes in 177 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 15:54:45.471573 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c665f3f9-7625-40ca-842e-a5ed13811620 None None] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:45.472104 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c665f3f9-7625-40ca-842e-a5ed13811620 None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:45.472104 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c665f3f9-7625-40ca-842e-a5ed13811620 None None] PATH_INFO: `/` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:45.473847 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 444/882] 10.222.0.50 () {54 vars in 927 bytes} [Tue Sep 13 15:54:45 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:54:45.480155 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b527b843-a7a9-4fbd-9761-2a7e6b81b29b None None] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:45.480654 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b527b843-a7a9-4fbd-9761-2a7e6b81b29b None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:45.480735 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b527b843-a7a9-4fbd-9761-2a7e6b81b29b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:45.499414 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.auth.core [None req-b527b843-a7a9-4fbd-9761-2a7e6b81b29b None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81438) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:54:45.500606 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-b527b843-a7a9-4fbd-9761-2a7e6b81b29b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:54:45.512333 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:54:45.582484 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 439/883] 10.222.0.50 () {58 vars in 1024 bytes} [Tue Sep 13 15:54:45 2022] POST /identity/v3/auth/tokens => generated 1372 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 13 15:54:45.854388 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c665f3f9-7625-40ca-842e-a5ed13811620 None None] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:45.952093 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:54:45.972040 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c665f3f9-7625-40ca-842e-a5ed13811620 None None] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:46.008748 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea99ceff-0361-4dfc-8677-64196b09e21a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:46.009387 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea99ceff-0361-4dfc-8677-64196b09e21a None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:46.009669 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea99ceff-0361-4dfc-8677-64196b09e21a None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:46.009954 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea99ceff-0361-4dfc-8677-64196b09e21a None admin] PATH_INFO: `/v3/services/volumev3` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:46.010539 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea99ceff-0361-4dfc-8677-64196b09e21a None admin] RBAC: Authorizing `identity:get_service(service_id=volumev3)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:46.011957 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea99ceff-0361-4dfc-8677-64196b09e21a None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:46.021110 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application [None req-ea99ceff-0361-4dfc-8677-64196b09e21a None admin] Could not find service: volumev3.: keystone.exception.ServiceNotFound: Could not find service: volumev3. Sep 13 15:54:46.021110 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:54:46.021110 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 13 15:54:46.021110 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 13 15:54:46.021110 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 13 15:54:46.021110 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 13 15:54:46.021110 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 13 15:54:46.021110 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 13 15:54:46.021110 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Sep 13 15:54:46.021110 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application Sep 13 15:54:46.021110 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 13 15:54:46.021110 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application Sep 13 15:54:46.021110 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:54:46.021110 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:54:46.021110 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:54:46.021110 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:54:46.021110 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:54:46.022088 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:54:46.022088 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:54:46.022088 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:54:46.022088 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:54:46.022088 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:54:46.022088 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:54:46.022088 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 13 15:54:46.022088 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 13 15:54:46.022088 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 13 15:54:46.022088 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 13 15:54:46.022088 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:54:46.022088 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:54:46.022088 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:54:46.022088 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:54:46.022088 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:54:46.022088 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:54:46.023135 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:54:46.023135 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application with Lock( Sep 13 15:54:46.023135 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:54:46.023135 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:54:46.023135 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:54:46.023135 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:54:46.023135 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:54:46.023135 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:54:46.023135 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:54:46.023135 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:54:46.023135 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 13 15:54:46.023135 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 13 15:54:46.023135 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Sep 13 15:54:46.023135 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application Sep 13 15:54:46.023135 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 445/884] 10.222.0.50 () {56 vars in 1114 bytes} [Tue Sep 13 15:54:45 2022] GET /identity/v3/services/volumev3 => generated 89 bytes in 168 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:54:46.030942 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b527b843-a7a9-4fbd-9761-2a7e6b81b29b None None] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:46.072625 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ae13312-135b-4f27-8166-c83e736b3023 None None] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:46.072923 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ae13312-135b-4f27-8166-c83e736b3023 None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:46.073205 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ae13312-135b-4f27-8166-c83e736b3023 None None] PATH_INFO: `/` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:46.074532 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 446/885] 10.222.0.50 () {54 vars in 927 bytes} [Tue Sep 13 15:54:46 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 13 15:54:46.080894 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d2339bd-342d-4d69-be20-7cc995974ee7 None None] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:46.081179 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d2339bd-342d-4d69-be20-7cc995974ee7 None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:46.081459 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d2339bd-342d-4d69-be20-7cc995974ee7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:46.100028 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.auth.core [None req-9d2339bd-342d-4d69-be20-7cc995974ee7 None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81437) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:54:46.101286 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-9d2339bd-342d-4d69-be20-7cc995974ee7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:54:46.156295 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b527b843-a7a9-4fbd-9761-2a7e6b81b29b None None] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:46.173878 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 447/886] 10.222.0.50 () {58 vars in 1024 bytes} [Tue Sep 13 15:54:46 2022] POST /identity/v3/auth/tokens => generated 1372 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 13 15:54:46.198605 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20daed07-34df-44f0-95a9-ae56a555b953 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:46.199501 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20daed07-34df-44f0-95a9-ae56a555b953 None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:46.199953 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20daed07-34df-44f0-95a9-ae56a555b953 None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:46.200380 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20daed07-34df-44f0-95a9-ae56a555b953 None admin] PATH_INFO: `/v3/services` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:46.201159 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20daed07-34df-44f0-95a9-ae56a555b953 None admin] RBAC: Adding query filter params (name=volumev3) {{(pid=81438) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:54:46.201608 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20daed07-34df-44f0-95a9-ae56a555b953 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:46.203213 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20daed07-34df-44f0-95a9-ae56a555b953 None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:46.213141 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 440/887] 10.222.0.50 () {56 vars in 1114 bytes} [Tue Sep 13 15:54:46 2022] GET /identity/v3/services?name=volumev3 => generated 125 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:54:46.214785 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d2339bd-342d-4d69-be20-7cc995974ee7 None None] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:46.300042 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:54:46.321838 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d2339bd-342d-4d69-be20-7cc995974ee7 None None] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:46.357049 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85c955af-2eb6-43d6-b1d5-f857b1ea09a9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:46.357661 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85c955af-2eb6-43d6-b1d5-f857b1ea09a9 None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:46.357943 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85c955af-2eb6-43d6-b1d5-f857b1ea09a9 None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:46.358222 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85c955af-2eb6-43d6-b1d5-f857b1ea09a9 None admin] PATH_INFO: `/v3/services` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:46.358935 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85c955af-2eb6-43d6-b1d5-f857b1ea09a9 None admin] RBAC: Adding query filter params (type=volumev3) {{(pid=81437) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:54:46.359253 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85c955af-2eb6-43d6-b1d5-f857b1ea09a9 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:46.360630 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85c955af-2eb6-43d6-b1d5-f857b1ea09a9 None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:46.365985 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 448/888] 10.222.0.50 () {56 vars in 1114 bytes} [Tue Sep 13 15:54:46 2022] GET /identity/v3/services?type=volumev3 => generated 362 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:54:46.373067 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20daed07-34df-44f0-95a9-ae56a555b953 None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:46.382470 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:54:46.447805 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85c955af-2eb6-43d6-b1d5-f857b1ea09a9 None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:46.488131 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20daed07-34df-44f0-95a9-ae56a555b953 None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:46.529936 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40a2b2b1-3737-4e7d-b948-e30019a76a3a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:46.530537 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40a2b2b1-3737-4e7d-b948-e30019a76a3a None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:46.531019 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40a2b2b1-3737-4e7d-b948-e30019a76a3a None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:46.531429 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40a2b2b1-3737-4e7d-b948-e30019a76a3a None admin] PATH_INFO: `/v3/endpoints` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:46.532284 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40a2b2b1-3737-4e7d-b948-e30019a76a3a None admin] RBAC: Adding query filter params (interface=public, service_id=be2c0958a4d2433a8c771dca86a3b099, region_id=RegionOne) {{(pid=81438) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:54:46.532746 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40a2b2b1-3737-4e7d-b948-e30019a76a3a None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:46.534243 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40a2b2b1-3737-4e7d-b948-e30019a76a3a None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:46.540574 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 441/889] 10.222.0.50 () {56 vars in 1251 bytes} [Tue Sep 13 15:54:46 2022] GET /identity/v3/endpoints?service_id=be2c0958a4d2433a8c771dca86a3b099&interface=public®ion_id=RegionOne => generated 194 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:54:46.545926 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40a2b2b1-3737-4e7d-b948-e30019a76a3a None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:46.569320 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85c955af-2eb6-43d6-b1d5-f857b1ea09a9 None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:46.608551 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8261e78c-5730-4d21-8db1-bfb2bfc72b47 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:46.609437 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8261e78c-5730-4d21-8db1-bfb2bfc72b47 None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:46.609893 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8261e78c-5730-4d21-8db1-bfb2bfc72b47 None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:46.610446 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8261e78c-5730-4d21-8db1-bfb2bfc72b47 None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:46.621473 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8261e78c-5730-4d21-8db1-bfb2bfc72b47 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 13 15:54:46.627060 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application [None req-8261e78c-5730-4d21-8db1-bfb2bfc72b47 None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Sep 13 15:54:46.627060 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:54:46.627060 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:54:46.627060 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:54:46.627060 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:54:46.627060 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:54:46.627060 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:54:46.627060 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:54:46.627060 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:54:46.627060 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:54:46.627060 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:54:46.627060 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:54:46.627060 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 13 15:54:46.627060 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 13 15:54:46.627060 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 13 15:54:46.627060 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 13 15:54:46.628023 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 13 15:54:46.628023 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 13 15:54:46.628023 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 13 15:54:46.628023 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 13 15:54:46.628023 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:54:46.628023 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:54:46.628023 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:54:46.628023 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:54:46.628023 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:54:46.628023 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:54:46.628023 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:54:46.628023 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application with Lock( Sep 13 15:54:46.628023 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:54:46.628023 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:54:46.628023 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:54:46.628023 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:54:46.628871 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:54:46.628871 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:54:46.628871 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:54:46.628871 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:54:46.628871 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 13 15:54:46.628871 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 13 15:54:46.628871 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 13 15:54:46.628871 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 13 15:54:46.628871 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 13 15:54:46.628871 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 13 15:54:46.628871 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Sep 13 15:54:46.628871 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application Sep 13 15:54:46.631297 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:54:46.637185 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 449/890] 10.222.0.50 () {56 vars in 1120 bytes} [Tue Sep 13 15:54:46 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 190 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 13 15:54:46.638336 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8261e78c-5730-4d21-8db1-bfb2bfc72b47 None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:46.641398 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:54:46.668587 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40a2b2b1-3737-4e7d-b948-e30019a76a3a None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:46.715744 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46bc1b29-ad90-4894-8ab0-066c424a7372 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:46.716306 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46bc1b29-ad90-4894-8ab0-066c424a7372 None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:46.716587 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46bc1b29-ad90-4894-8ab0-066c424a7372 None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:46.716860 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46bc1b29-ad90-4894-8ab0-066c424a7372 None admin] PATH_INFO: `/v3/services` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:46.717462 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46bc1b29-ad90-4894-8ab0-066c424a7372 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:46.719666 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46bc1b29-ad90-4894-8ab0-066c424a7372 None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:46.725921 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 442/891] 10.222.0.50 () {56 vars in 1087 bytes} [Tue Sep 13 15:54:46 2022] GET /identity/v3/services => generated 1268 bytes in 180 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 13 15:54:46.766994 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8261e78c-5730-4d21-8db1-bfb2bfc72b47 None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:46.812969 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-013112ea-8526-4bb6-9ad0-a809cb7f9f40 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:46.814233 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-013112ea-8526-4bb6-9ad0-a809cb7f9f40 None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:46.814810 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-013112ea-8526-4bb6-9ad0-a809cb7f9f40 None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:46.815349 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-013112ea-8526-4bb6-9ad0-a809cb7f9f40 None admin] PATH_INFO: `/v3/roles` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:46.816341 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-013112ea-8526-4bb6-9ad0-a809cb7f9f40 None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=81437) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:54:46.816908 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-013112ea-8526-4bb6-9ad0-a809cb7f9f40 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:46.818553 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-013112ea-8526-4bb6-9ad0-a809cb7f9f40 None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:46.829824 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 450/892] 10.222.0.50 () {56 vars in 1115 bytes} [Tue Sep 13 15:54:46 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 192 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:54:46.837157 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46bc1b29-ad90-4894-8ab0-066c424a7372 None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:46.915157 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2f04dbb-92f2-4e05-9561-92897c059ec1 None None] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:46.915773 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2f04dbb-92f2-4e05-9561-92897c059ec1 None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:46.916288 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2f04dbb-92f2-4e05-9561-92897c059ec1 None None] PATH_INFO: `/` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:46.917940 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 451/893] 10.222.0.50 () {54 vars in 927 bytes} [Tue Sep 13 15:54:46 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 13 15:54:46.923686 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa3a2606-33b7-458c-80b6-cb895673f05d None None] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:46.923686 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa3a2606-33b7-458c-80b6-cb895673f05d None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:46.923922 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa3a2606-33b7-458c-80b6-cb895673f05d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:46.946704 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.auth.core [None req-aa3a2606-33b7-458c-80b6-cb895673f05d None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81437) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:54:46.948042 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-aa3a2606-33b7-458c-80b6-cb895673f05d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:54:46.956692 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46bc1b29-ad90-4894-8ab0-066c424a7372 None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:46.994219 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91f10bed-1e26-42dd-bb52-86fb1afbd5ab None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:46.994785 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91f10bed-1e26-42dd-bb52-86fb1afbd5ab None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:46.995098 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91f10bed-1e26-42dd-bb52-86fb1afbd5ab None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:46.995736 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91f10bed-1e26-42dd-bb52-86fb1afbd5ab None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:46.998208 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:54:47.009365 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91f10bed-1e26-42dd-bb52-86fb1afbd5ab None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:47.010821 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91f10bed-1e26-42dd-bb52-86fb1afbd5ab None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:47.014302 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:54:47.030569 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 452/894] 10.222.0.50 () {58 vars in 1024 bytes} [Tue Sep 13 15:54:46 2022] POST /identity/v3/auth/tokens => generated 1372 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 13 15:54:47.030569 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1406cb71-28ee-45ae-b727-2e809a101681 None None] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:47.030569 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1406cb71-28ee-45ae-b727-2e809a101681 None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:47.030569 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1406cb71-28ee-45ae-b727-2e809a101681 None None] PATH_INFO: `/` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:47.030569 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 453/895] 10.222.0.50 () {54 vars in 927 bytes} [Tue Sep 13 15:54:47 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:54:47.031083 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application [None req-91f10bed-1e26-42dd-bb52-86fb1afbd5ab None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:54:47.031083 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:54:47.031083 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:54:47.031083 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:54:47.031083 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:54:47.031083 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:54:47.031083 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:54:47.031083 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:54:47.031083 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:54:47.031083 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:54:47.031083 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:54:47.031083 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:54:47.031083 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 13 15:54:47.031083 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 13 15:54:47.031083 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 13 15:54:47.031083 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 13 15:54:47.031970 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:54:47.031970 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:54:47.031970 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:54:47.031970 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:54:47.031970 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:54:47.031970 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:54:47.031970 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:54:47.031970 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application with Lock( Sep 13 15:54:47.031970 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:54:47.031970 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:54:47.031970 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:54:47.031970 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:54:47.031970 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:54:47.031970 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:54:47.031970 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:54:47.031970 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:54:47.031970 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 13 15:54:47.032888 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 13 15:54:47.032888 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:54:47.032888 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Sep 13 15:54:47.032888 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 443/896] 10.222.0.50 () {56 vars in 1108 bytes} [Tue Sep 13 15:54:46 2022] GET /identity/v3/domains/Default => generated 87 bytes in 192 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:54:47.033118 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75674458-e8d3-4a9a-9c09-ec88ff8214fa None None] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:47.033118 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75674458-e8d3-4a9a-9c09-ec88ff8214fa None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:47.033118 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75674458-e8d3-4a9a-9c09-ec88ff8214fa None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:47.043934 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91f10bed-1e26-42dd-bb52-86fb1afbd5ab None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:47.056125 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.auth.core [None req-75674458-e8d3-4a9a-9c09-ec88ff8214fa None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81437) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:54:47.057678 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-75674458-e8d3-4a9a-9c09-ec88ff8214fa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:54:47.139085 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 454/897] 10.222.0.50 () {58 vars in 1024 bytes} [Tue Sep 13 15:54:47 2022] POST /identity/v3/auth/tokens => generated 1372 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 13 15:54:47.168634 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91f10bed-1e26-42dd-bb52-86fb1afbd5ab None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:47.204041 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4431d428-46ea-40ca-aecd-3c8f9364e501 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:47.204911 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4431d428-46ea-40ca-aecd-3c8f9364e501 None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:47.205362 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4431d428-46ea-40ca-aecd-3c8f9364e501 None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:47.205773 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4431d428-46ea-40ca-aecd-3c8f9364e501 None admin] PATH_INFO: `/v3/domains` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:47.206551 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4431d428-46ea-40ca-aecd-3c8f9364e501 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81438) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:54:47.207017 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4431d428-46ea-40ca-aecd-3c8f9364e501 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:47.208520 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4431d428-46ea-40ca-aecd-3c8f9364e501 None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:47.221837 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 444/898] 10.222.0.50 () {56 vars in 1109 bytes} [Tue Sep 13 15:54:47 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:54:47.227687 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75674458-e8d3-4a9a-9c09-ec88ff8214fa None None] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:47.297646 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4431d428-46ea-40ca-aecd-3c8f9364e501 None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:47.337305 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75674458-e8d3-4a9a-9c09-ec88ff8214fa None None] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:47.352777 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:54:47.363625 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:54:47.384602 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60c30d13-dc00-46b7-b1f5-1654de0acc13 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:47.385446 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60c30d13-dc00-46b7-b1f5-1654de0acc13 None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:47.385888 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60c30d13-dc00-46b7-b1f5-1654de0acc13 None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:47.386303 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60c30d13-dc00-46b7-b1f5-1654de0acc13 None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:47.394025 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60c30d13-dc00-46b7-b1f5-1654de0acc13 None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:47.395777 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60c30d13-dc00-46b7-b1f5-1654de0acc13 None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:47.408932 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application [None req-60c30d13-dc00-46b7-b1f5-1654de0acc13 None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Sep 13 15:54:47.408932 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:54:47.408932 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:54:47.408932 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:54:47.408932 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:54:47.408932 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:54:47.408932 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:54:47.408932 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:54:47.408932 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:54:47.408932 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:54:47.408932 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:54:47.408932 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:54:47.408932 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 13 15:54:47.408932 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 13 15:54:47.408932 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 13 15:54:47.408932 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 13 15:54:47.409902 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:54:47.409902 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:54:47.409902 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 13 15:54:47.409902 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 13 15:54:47.409902 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 13 15:54:47.409902 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 13 15:54:47.409902 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:54:47.409902 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:54:47.409902 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:54:47.409902 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:54:47.409902 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:54:47.409902 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application with Lock( Sep 13 15:54:47.409902 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:54:47.409902 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:54:47.409902 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:54:47.409902 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:54:47.409902 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:54:47.409902 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:54:47.410924 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:54:47.410924 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:54:47.410924 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 13 15:54:47.410924 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 13 15:54:47.410924 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 13 15:54:47.410924 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 13 15:54:47.410924 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 13 15:54:47.410924 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 13 15:54:47.410924 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Sep 13 15:54:47.410924 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application Sep 13 15:54:47.418199 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 455/899] 10.222.0.50 () {56 vars in 1117 bytes} [Tue Sep 13 15:54:47 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 186 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:54:47.419376 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60c30d13-dc00-46b7-b1f5-1654de0acc13 None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:47.427695 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4431d428-46ea-40ca-aecd-3c8f9364e501 None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:47.470646 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83d799f6-2860-4dc6-a3ee-d517fdf2f190 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:47.471300 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83d799f6-2860-4dc6-a3ee-d517fdf2f190 None admin] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:47.471594 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83d799f6-2860-4dc6-a3ee-d517fdf2f190 None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:47.471855 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83d799f6-2860-4dc6-a3ee-d517fdf2f190 None admin] PATH_INFO: `/v3/services` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:47.472514 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83d799f6-2860-4dc6-a3ee-d517fdf2f190 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:47.473876 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83d799f6-2860-4dc6-a3ee-d517fdf2f190 None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:47.493335 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 445/900] 10.222.0.50 () {60 vars in 1141 bytes} [Tue Sep 13 15:54:47 2022] POST /identity/v3/services => generated 241 bytes in 196 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 13 15:54:47.496098 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83d799f6-2860-4dc6-a3ee-d517fdf2f190 None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:47.532589 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60c30d13-dc00-46b7-b1f5-1654de0acc13 None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:47.580441 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0a77d04-9749-4281-a3e1-532edc490e9f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:47.581992 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0a77d04-9749-4281-a3e1-532edc490e9f None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:47.582860 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0a77d04-9749-4281-a3e1-532edc490e9f None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:47.583325 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0a77d04-9749-4281-a3e1-532edc490e9f None admin] PATH_INFO: `/v3/users` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:47.584227 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0a77d04-9749-4281-a3e1-532edc490e9f None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=81437) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:54:47.584706 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0a77d04-9749-4281-a3e1-532edc490e9f None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:47.587712 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0a77d04-9749-4281-a3e1-532edc490e9f None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:47.620924 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 456/901] 10.222.0.50 () {56 vars in 1149 bytes} [Tue Sep 13 15:54:47 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 202 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:54:47.622486 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0a77d04-9749-4281-a3e1-532edc490e9f None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:47.647803 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83d799f6-2860-4dc6-a3ee-d517fdf2f190 None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:47.657591 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:54:47.685060 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:54:47.700770 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2c5fe74-1e86-4db5-987e-3129adac8c4d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:47.701621 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2c5fe74-1e86-4db5-987e-3129adac8c4d None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:47.702032 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2c5fe74-1e86-4db5-987e-3129adac8c4d None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:47.702429 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2c5fe74-1e86-4db5-987e-3129adac8c4d None admin] PATH_INFO: `/v3/services/compute_legacy` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:47.703741 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2c5fe74-1e86-4db5-987e-3129adac8c4d None admin] RBAC: Authorizing `identity:get_service(service_id=compute_legacy)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:47.705853 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2c5fe74-1e86-4db5-987e-3129adac8c4d None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:47.719000 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application [None req-f2c5fe74-1e86-4db5-987e-3129adac8c4d None admin] Could not find service: compute_legacy.: keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Sep 13 15:54:47.719000 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:54:47.719000 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 13 15:54:47.719000 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 13 15:54:47.719000 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 13 15:54:47.719000 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 13 15:54:47.719000 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 13 15:54:47.719000 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 13 15:54:47.719000 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Sep 13 15:54:47.719000 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Sep 13 15:54:47.719000 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 13 15:54:47.719000 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Sep 13 15:54:47.719000 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:54:47.719000 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:54:47.719000 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:54:47.719000 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:54:47.720139 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:54:47.720139 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:54:47.720139 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:54:47.720139 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:54:47.720139 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:54:47.720139 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:54:47.720139 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:54:47.720139 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 13 15:54:47.720139 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 13 15:54:47.720139 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 13 15:54:47.720139 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 13 15:54:47.720139 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:54:47.720139 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:54:47.720139 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:54:47.720139 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:54:47.720139 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:54:47.720139 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:54:47.721214 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:54:47.721214 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application with Lock( Sep 13 15:54:47.721214 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:54:47.721214 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:54:47.721214 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:54:47.721214 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:54:47.721214 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:54:47.721214 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:54:47.721214 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:54:47.721214 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:54:47.721214 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 13 15:54:47.721214 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 13 15:54:47.721214 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Sep 13 15:54:47.721214 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Sep 13 15:54:47.721214 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 446/902] 10.222.0.50 () {56 vars in 1132 bytes} [Tue Sep 13 15:54:47 2022] GET /identity/v3/services/compute_legacy => generated 95 bytes in 224 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:54:47.724675 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2c5fe74-1e86-4db5-987e-3129adac8c4d None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:47.762240 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0a77d04-9749-4281-a3e1-532edc490e9f None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:47.800298 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-826e0745-e788-4b19-abbb-c4b4e0d5fec4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:47.800912 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-826e0745-e788-4b19-abbb-c4b4e0d5fec4 None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:47.801193 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-826e0745-e788-4b19-abbb-c4b4e0d5fec4 None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:47.801468 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-826e0745-e788-4b19-abbb-c4b4e0d5fec4 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:47.857343 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2c5fe74-1e86-4db5-987e-3129adac8c4d None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:47.888562 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-826e0745-e788-4b19-abbb-c4b4e0d5fec4 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:47.890305 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-826e0745-e788-4b19-abbb-c4b4e0d5fec4 None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:47.896873 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c357ff3-8b7c-4d4b-a981-f80d7b7a2b68 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:47.897762 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c357ff3-8b7c-4d4b-a981-f80d7b7a2b68 None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:47.898306 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c357ff3-8b7c-4d4b-a981-f80d7b7a2b68 None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:47.898945 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c357ff3-8b7c-4d4b-a981-f80d7b7a2b68 None admin] PATH_INFO: `/v3/services` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:47.899849 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c357ff3-8b7c-4d4b-a981-f80d7b7a2b68 None admin] RBAC: Adding query filter params (name=compute_legacy) {{(pid=81438) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:54:47.900473 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c357ff3-8b7c-4d4b-a981-f80d7b7a2b68 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:47.901928 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c357ff3-8b7c-4d4b-a981-f80d7b7a2b68 None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:47.912455 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 447/903] 10.222.0.50 () {56 vars in 1126 bytes} [Tue Sep 13 15:54:47 2022] GET /identity/v3/services?name=compute_legacy => generated 131 bytes in 183 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 13 15:54:47.913540 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c357ff3-8b7c-4d4b-a981-f80d7b7a2b68 None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:47.952101 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:54:47.955404 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:54:48.026721 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 457/904] 10.222.0.50 () {58 vars in 1282 bytes} [Tue Sep 13 15:54:47 2022] GET /identity/v3/auth/tokens => generated 1474 bytes in 405 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 13 15:54:48.029070 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c357ff3-8b7c-4d4b-a981-f80d7b7a2b68 None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:48.040568 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-826e0745-e788-4b19-abbb-c4b4e0d5fec4 None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:48.072827 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8da88198-ca15-4f5a-85dc-d7291c3a1e22 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:48.073610 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8da88198-ca15-4f5a-85dc-d7291c3a1e22 None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:48.074065 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8da88198-ca15-4f5a-85dc-d7291c3a1e22 None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:48.074431 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8da88198-ca15-4f5a-85dc-d7291c3a1e22 None admin] PATH_INFO: `/v3/services` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:48.075353 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8da88198-ca15-4f5a-85dc-d7291c3a1e22 None admin] RBAC: Adding query filter params (type=compute_legacy) {{(pid=81438) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:54:48.075782 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8da88198-ca15-4f5a-85dc-d7291c3a1e22 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:48.078218 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8da88198-ca15-4f5a-85dc-d7291c3a1e22 None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:48.089542 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 448/905] 10.222.0.50 () {56 vars in 1126 bytes} [Tue Sep 13 15:54:47 2022] GET /identity/v3/services?type=compute_legacy => generated 386 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 13 15:54:48.090702 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8da88198-ca15-4f5a-85dc-d7291c3a1e22 None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:48.155494 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-826e0745-e788-4b19-abbb-c4b4e0d5fec4 None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:48.192191 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-240ff0d5-bf0c-4434-aa17-96d524d5eb3a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:48.193209 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-240ff0d5-bf0c-4434-aa17-96d524d5eb3a None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:48.193761 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-240ff0d5-bf0c-4434-aa17-96d524d5eb3a None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:48.194344 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-240ff0d5-bf0c-4434-aa17-96d524d5eb3a None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:48.203150 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:54:48.208392 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-240ff0d5-bf0c-4434-aa17-96d524d5eb3a None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:48.210104 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-240ff0d5-bf0c-4434-aa17-96d524d5eb3a None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:48.214349 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:54:48.218118 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8da88198-ca15-4f5a-85dc-d7291c3a1e22 None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:48.228388 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application [None req-240ff0d5-bf0c-4434-aa17-96d524d5eb3a None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:54:48.228388 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:54:48.228388 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:54:48.228388 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:54:48.228388 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:54:48.228388 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:54:48.228388 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:54:48.228388 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:54:48.228388 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:54:48.228388 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:54:48.228388 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:54:48.228388 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:54:48.228388 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 13 15:54:48.228388 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 13 15:54:48.228388 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 13 15:54:48.228388 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 13 15:54:48.229391 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:54:48.229391 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:54:48.229391 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:54:48.229391 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:54:48.229391 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:54:48.229391 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:54:48.229391 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:54:48.229391 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application with Lock( Sep 13 15:54:48.229391 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:54:48.229391 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:54:48.229391 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:54:48.229391 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:54:48.229391 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:54:48.229391 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:54:48.229391 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:54:48.229391 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:54:48.229391 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 13 15:54:48.230311 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 13 15:54:48.230311 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:54:48.230311 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application Sep 13 15:54:48.232558 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 458/906] 10.222.0.50 () {56 vars in 1108 bytes} [Tue Sep 13 15:54:48 2022] GET /identity/v3/domains/Default => generated 87 bytes in 193 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:54:48.238949 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-240ff0d5-bf0c-4434-aa17-96d524d5eb3a None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:48.261934 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fd9f8eb-e91f-4ce1-9dbe-84d827c98baf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:48.262557 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fd9f8eb-e91f-4ce1-9dbe-84d827c98baf None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:48.262841 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fd9f8eb-e91f-4ce1-9dbe-84d827c98baf None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:48.263147 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fd9f8eb-e91f-4ce1-9dbe-84d827c98baf None admin] PATH_INFO: `/v3/endpoints` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:48.263849 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fd9f8eb-e91f-4ce1-9dbe-84d827c98baf None admin] RBAC: Adding query filter params (interface=public, service_id=3298ef47490d4627bb59c20f0ff8a397, region_id=RegionOne) {{(pid=81438) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:54:48.264429 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fd9f8eb-e91f-4ce1-9dbe-84d827c98baf None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:48.266324 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fd9f8eb-e91f-4ce1-9dbe-84d827c98baf None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:48.272895 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 449/907] 10.222.0.50 () {56 vars in 1251 bytes} [Tue Sep 13 15:54:48 2022] GET /identity/v3/endpoints?service_id=3298ef47490d4627bb59c20f0ff8a397&interface=public®ion_id=RegionOne => generated 194 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:54:48.281730 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fd9f8eb-e91f-4ce1-9dbe-84d827c98baf None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:48.377110 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-240ff0d5-bf0c-4434-aa17-96d524d5eb3a None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:48.394011 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fd9f8eb-e91f-4ce1-9dbe-84d827c98baf None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:48.423297 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-389fc2ef-8f80-4d63-9076-27b35eb970ef None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:48.423954 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-389fc2ef-8f80-4d63-9076-27b35eb970ef None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:48.424530 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-389fc2ef-8f80-4d63-9076-27b35eb970ef None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:48.424866 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-389fc2ef-8f80-4d63-9076-27b35eb970ef None admin] PATH_INFO: `/v3/domains` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:48.425985 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-389fc2ef-8f80-4d63-9076-27b35eb970ef None admin] RBAC: Adding query filter params (name=Default) {{(pid=81437) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:54:48.426644 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-389fc2ef-8f80-4d63-9076-27b35eb970ef None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:48.428788 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-389fc2ef-8f80-4d63-9076-27b35eb970ef None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:48.434470 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37ac57e5-c7ab-406b-ad23-3a7141fb3b16 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:48.435206 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37ac57e5-c7ab-406b-ad23-3a7141fb3b16 None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:48.435562 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37ac57e5-c7ab-406b-ad23-3a7141fb3b16 None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:48.435911 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37ac57e5-c7ab-406b-ad23-3a7141fb3b16 None admin] PATH_INFO: `/v3/services` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:48.436587 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37ac57e5-c7ab-406b-ad23-3a7141fb3b16 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:48.438012 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37ac57e5-c7ab-406b-ad23-3a7141fb3b16 None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:48.444302 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 459/908] 10.222.0.50 () {56 vars in 1109 bytes} [Tue Sep 13 15:54:48 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 207 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 13 15:54:48.449306 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 450/909] 10.222.0.50 () {56 vars in 1087 bytes} [Tue Sep 13 15:54:48 2022] GET /identity/v3/services => generated 1497 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 13 15:54:48.451270 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-389fc2ef-8f80-4d63-9076-27b35eb970ef None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:48.515546 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:54:48.579644 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-389fc2ef-8f80-4d63-9076-27b35eb970ef None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:48.620948 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-207e6be0-4dcf-48db-a794-01c2ca21c55f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:48.622466 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-207e6be0-4dcf-48db-a794-01c2ca21c55f None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:48.623004 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-207e6be0-4dcf-48db-a794-01c2ca21c55f None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:48.623489 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-207e6be0-4dcf-48db-a794-01c2ca21c55f None admin] PATH_INFO: `/v3/projects/service` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:48.631650 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-207e6be0-4dcf-48db-a794-01c2ca21c55f None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:48.634804 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-207e6be0-4dcf-48db-a794-01c2ca21c55f None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:48.652826 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application [None req-207e6be0-4dcf-48db-a794-01c2ca21c55f None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 13 15:54:48.652826 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:54:48.652826 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:54:48.652826 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:54:48.652826 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:54:48.652826 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:54:48.652826 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:54:48.652826 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:54:48.652826 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:54:48.652826 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:54:48.652826 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:54:48.652826 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:54:48.652826 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 13 15:54:48.652826 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 13 15:54:48.652826 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 13 15:54:48.653773 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 13 15:54:48.653773 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:54:48.653773 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:54:48.653773 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:54:48.653773 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:54:48.653773 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:54:48.653773 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:54:48.653773 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:54:48.653773 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application with Lock( Sep 13 15:54:48.653773 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:54:48.653773 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:54:48.653773 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:54:48.653773 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:54:48.653773 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:54:48.653773 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:54:48.653773 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:54:48.653773 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:54:48.655255 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 13 15:54:48.655255 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 13 15:54:48.655255 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 13 15:54:48.655255 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 13 15:54:48.655255 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 13 15:54:48.655255 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 13 15:54:48.655255 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 13 15:54:48.655255 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application Sep 13 15:54:48.655255 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 460/910] 10.222.0.50 () {56 vars in 1111 bytes} [Tue Sep 13 15:54:48 2022] GET /identity/v3/projects/service => generated 88 bytes in 203 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:54:48.660018 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37ac57e5-c7ab-406b-ad23-3a7141fb3b16 None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:48.693097 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:54:48.778686 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37ac57e5-c7ab-406b-ad23-3a7141fb3b16 None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:48.817587 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cffb8e37-ff94-44a0-a02c-e6c20b5dd9fe None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:48.818572 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cffb8e37-ff94-44a0-a02c-e6c20b5dd9fe None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:48.819093 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cffb8e37-ff94-44a0-a02c-e6c20b5dd9fe None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:48.819579 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cffb8e37-ff94-44a0-a02c-e6c20b5dd9fe None admin] PATH_INFO: `/v3/projects` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:48.821094 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cffb8e37-ff94-44a0-a02c-e6c20b5dd9fe None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81438) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:54:48.821690 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cffb8e37-ff94-44a0-a02c-e6c20b5dd9fe None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:48.823349 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cffb8e37-ff94-44a0-a02c-e6c20b5dd9fe None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:48.837305 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 451/911] 10.222.0.50 () {56 vars in 1148 bytes} [Tue Sep 13 15:54:48 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 13 15:54:48.843419 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-207e6be0-4dcf-48db-a794-01c2ca21c55f None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:48.962202 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-207e6be0-4dcf-48db-a794-01c2ca21c55f None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:48.967713 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3188b62-a840-448d-bd40-5495f8ae30f9 None None] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:48.968255 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3188b62-a840-448d-bd40-5495f8ae30f9 None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:48.968736 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3188b62-a840-448d-bd40-5495f8ae30f9 None None] PATH_INFO: `/` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:48.970559 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 452/912] 10.222.0.50 () {54 vars in 927 bytes} [Tue Sep 13 15:54:48 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 13 15:54:48.976930 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cf95d4c-7d16-4805-b06e-8aefd8834b88 None None] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:48.977812 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cf95d4c-7d16-4805-b06e-8aefd8834b88 None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:48.978314 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cf95d4c-7d16-4805-b06e-8aefd8834b88 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:48.978850 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:54:49.001911 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.auth.core [None req-8cf95d4c-7d16-4805-b06e-8aefd8834b88 None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81438) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:54:49.003246 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-8cf95d4c-7d16-4805-b06e-8aefd8834b88 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:54:49.010491 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eda67593-3bef-4301-bd87-9300c9b0f3dd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:49.011422 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eda67593-3bef-4301-bd87-9300c9b0f3dd None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:49.011911 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eda67593-3bef-4301-bd87-9300c9b0f3dd None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:49.012981 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eda67593-3bef-4301-bd87-9300c9b0f3dd None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:49.013941 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eda67593-3bef-4301-bd87-9300c9b0f3dd None admin] RBAC: Adding query filter params (role.id=20e2f11234344900a51fc76fe5b625f3, scope.project.id=5811e3b83a1e462e9320cad79a3ad791, user.id=f8364b1a190d4218bf353a89bdb81448) {{(pid=81437) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:54:49.014511 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eda67593-3bef-4301-bd87-9300c9b0f3dd None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:49.017086 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eda67593-3bef-4301-bd87-9300c9b0f3dd None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:49.025494 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 461/913] 10.222.0.50 () {56 vars in 1374 bytes} [Tue Sep 13 15:54:48 2022] GET /identity/v3/role_assignments?user.id=f8364b1a190d4218bf353a89bdb81448&scope.project.id=5811e3b83a1e462e9320cad79a3ad791&role.id=20e2f11234344900a51fc76fe5b625f3 => generated 259 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:54:49.099726 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 453/914] 10.222.0.50 () {58 vars in 1024 bytes} [Tue Sep 13 15:54:48 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 123 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 13 15:54:49.373605 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eda67593-3bef-4301-bd87-9300c9b0f3dd None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:49.481347 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eda67593-3bef-4301-bd87-9300c9b0f3dd None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:49.517151 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d30f2aa1-7b27-4563-8568-8ca92d69b946 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:49.518131 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d30f2aa1-7b27-4563-8568-8ca92d69b946 None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:49.518607 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d30f2aa1-7b27-4563-8568-8ca92d69b946 None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:49.519107 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d30f2aa1-7b27-4563-8568-8ca92d69b946 None admin] PATH_INFO: `/v3/services/volumev3` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:49.519864 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d30f2aa1-7b27-4563-8568-8ca92d69b946 None admin] RBAC: Authorizing `identity:get_service(service_id=volumev3)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:49.521407 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d30f2aa1-7b27-4563-8568-8ca92d69b946 None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:49.530712 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application [None req-d30f2aa1-7b27-4563-8568-8ca92d69b946 None admin] Could not find service: volumev3.: keystone.exception.ServiceNotFound: Could not find service: volumev3. Sep 13 15:54:49.530712 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:54:49.530712 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 13 15:54:49.530712 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 13 15:54:49.530712 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 13 15:54:49.530712 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 13 15:54:49.530712 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 13 15:54:49.530712 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 13 15:54:49.530712 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Sep 13 15:54:49.530712 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application Sep 13 15:54:49.530712 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 13 15:54:49.530712 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application Sep 13 15:54:49.530712 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:54:49.530712 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:54:49.530712 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:54:49.530712 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:54:49.530712 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:54:49.531756 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:54:49.531756 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:54:49.531756 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:54:49.531756 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:54:49.531756 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:54:49.531756 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:54:49.531756 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 13 15:54:49.531756 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 13 15:54:49.531756 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 13 15:54:49.531756 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 13 15:54:49.531756 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:54:49.531756 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:54:49.531756 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:54:49.531756 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:54:49.531756 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:54:49.531756 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:54:49.532604 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:54:49.532604 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application with Lock( Sep 13 15:54:49.532604 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:54:49.532604 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:54:49.532604 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:54:49.532604 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:54:49.532604 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:54:49.532604 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:54:49.532604 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:54:49.532604 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:54:49.532604 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 13 15:54:49.532604 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 13 15:54:49.532604 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Sep 13 15:54:49.532604 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application Sep 13 15:54:49.539402 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 462/915] 10.222.0.50 () {56 vars in 1114 bytes} [Tue Sep 13 15:54:49 2022] GET /identity/v3/services/volumev3 => generated 89 bytes in 162 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:54:49.541882 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cf95d4c-7d16-4805-b06e-8aefd8834b88 None None] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:49.564319 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:54:49.649680 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cf95d4c-7d16-4805-b06e-8aefd8834b88 None None] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:49.686768 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce865cef-751e-4098-a157-2eb32228d322 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:49.688309 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce865cef-751e-4098-a157-2eb32228d322 None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:49.688770 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce865cef-751e-4098-a157-2eb32228d322 None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:49.689225 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce865cef-751e-4098-a157-2eb32228d322 None admin] PATH_INFO: `/v3/services` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:49.690059 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce865cef-751e-4098-a157-2eb32228d322 None admin] RBAC: Adding query filter params (name=volumev3) {{(pid=81438) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:54:49.690588 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce865cef-751e-4098-a157-2eb32228d322 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:49.692220 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce865cef-751e-4098-a157-2eb32228d322 None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:49.702297 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 454/916] 10.222.0.50 () {56 vars in 1114 bytes} [Tue Sep 13 15:54:49 2022] GET /identity/v3/services?name=volumev3 => generated 125 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:54:49.703938 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d30f2aa1-7b27-4563-8568-8ca92d69b946 None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:49.748262 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:54:49.816890 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d30f2aa1-7b27-4563-8568-8ca92d69b946 None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:49.854425 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7676d2ba-2b49-4384-bfae-9d193ae06667 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:49.855377 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7676d2ba-2b49-4384-bfae-9d193ae06667 None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:49.855846 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7676d2ba-2b49-4384-bfae-9d193ae06667 None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:49.856300 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7676d2ba-2b49-4384-bfae-9d193ae06667 None admin] PATH_INFO: `/v3/services` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:49.857130 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7676d2ba-2b49-4384-bfae-9d193ae06667 None admin] RBAC: Adding query filter params (type=volumev3) {{(pid=81437) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:54:49.857623 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7676d2ba-2b49-4384-bfae-9d193ae06667 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:49.859197 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7676d2ba-2b49-4384-bfae-9d193ae06667 None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:49.869515 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 463/917] 10.222.0.50 () {56 vars in 1114 bytes} [Tue Sep 13 15:54:49 2022] GET /identity/v3/services?type=volumev3 => generated 362 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:54:49.871462 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce865cef-751e-4098-a157-2eb32228d322 None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:49.872447 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b42ba63e-4856-4413-9ad1-4ac30b7865a6 None None] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:49.872752 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b42ba63e-4856-4413-9ad1-4ac30b7865a6 None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:49.873001 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b42ba63e-4856-4413-9ad1-4ac30b7865a6 None None] PATH_INFO: `/` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:49.874520 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 464/918] 10.222.0.50 () {54 vars in 927 bytes} [Tue Sep 13 15:54:49 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:54:49.879714 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0429bd2-4231-44a8-bb2e-6495abbda1cc None None] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:49.879995 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0429bd2-4231-44a8-bb2e-6495abbda1cc None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:49.880273 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0429bd2-4231-44a8-bb2e-6495abbda1cc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:49.899234 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.auth.core [None req-f0429bd2-4231-44a8-bb2e-6495abbda1cc None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81437) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:54:49.901027 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-f0429bd2-4231-44a8-bb2e-6495abbda1cc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:54:49.965125 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:54:49.979275 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 465/919] 10.222.0.50 () {58 vars in 1024 bytes} [Tue Sep 13 15:54:49 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 13 15:54:49.990919 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce865cef-751e-4098-a157-2eb32228d322 None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:50.020213 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-666144a1-6f5d-4a3a-96b3-109035f0c8c7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:50.020814 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-666144a1-6f5d-4a3a-96b3-109035f0c8c7 None admin] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:50.021091 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-666144a1-6f5d-4a3a-96b3-109035f0c8c7 None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:50.021357 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-666144a1-6f5d-4a3a-96b3-109035f0c8c7 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:50.022021 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-666144a1-6f5d-4a3a-96b3-109035f0c8c7 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:50.023469 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-666144a1-6f5d-4a3a-96b3-109035f0c8c7 None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:50.051328 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 455/920] 10.222.0.50 () {60 vars in 1144 bytes} [Tue Sep 13 15:54:49 2022] POST /identity/v3/endpoints => generated 346 bytes in 175 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 15:54:50.254617 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0429bd2-4231-44a8-bb2e-6495abbda1cc None None] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:50.286340 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:54:50.362086 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0429bd2-4231-44a8-bb2e-6495abbda1cc None None] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:50.404448 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43d3f089-7837-4ac9-b761-ab6d36c182c9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:50.405166 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43d3f089-7837-4ac9-b761-ab6d36c182c9 None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:50.405453 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43d3f089-7837-4ac9-b761-ab6d36c182c9 None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:50.405730 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43d3f089-7837-4ac9-b761-ab6d36c182c9 None admin] PATH_INFO: `/v3/services/object-store` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:50.406320 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43d3f089-7837-4ac9-b761-ab6d36c182c9 None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:50.407741 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43d3f089-7837-4ac9-b761-ab6d36c182c9 None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:50.418030 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application [None req-43d3f089-7837-4ac9-b761-ab6d36c182c9 None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Sep 13 15:54:50.418030 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:54:50.418030 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 13 15:54:50.418030 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 13 15:54:50.418030 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 13 15:54:50.418030 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 13 15:54:50.418030 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 13 15:54:50.418030 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 13 15:54:50.418030 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Sep 13 15:54:50.418030 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application Sep 13 15:54:50.418030 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 13 15:54:50.418030 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application Sep 13 15:54:50.418030 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:54:50.418030 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:54:50.418030 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:54:50.418030 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:54:50.418030 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:54:50.419105 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:54:50.419105 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:54:50.419105 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:54:50.419105 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:54:50.419105 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:54:50.419105 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:54:50.419105 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 13 15:54:50.419105 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 13 15:54:50.419105 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 13 15:54:50.419105 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 13 15:54:50.419105 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:54:50.419105 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:54:50.419105 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:54:50.419105 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:54:50.419105 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:54:50.419105 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:54:50.419972 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:54:50.419972 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application with Lock( Sep 13 15:54:50.419972 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:54:50.419972 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:54:50.419972 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:54:50.419972 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:54:50.419972 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:54:50.419972 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:54:50.419972 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:54:50.419972 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:54:50.419972 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 13 15:54:50.419972 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 13 15:54:50.419972 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Sep 13 15:54:50.419972 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application Sep 13 15:54:50.419972 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 466/921] 10.222.0.50 () {56 vars in 1126 bytes} [Tue Sep 13 15:54:50 2022] GET /identity/v3/services/object-store => generated 93 bytes in 165 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:54:50.425584 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-666144a1-6f5d-4a3a-96b3-109035f0c8c7 None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:50.534302 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-666144a1-6f5d-4a3a-96b3-109035f0c8c7 None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:50.569181 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d38cabd-f247-4879-bb63-c56e6bdefe14 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:50.569788 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d38cabd-f247-4879-bb63-c56e6bdefe14 None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:50.570080 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d38cabd-f247-4879-bb63-c56e6bdefe14 None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:50.570358 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d38cabd-f247-4879-bb63-c56e6bdefe14 None admin] PATH_INFO: `/v3/services` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:50.571104 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d38cabd-f247-4879-bb63-c56e6bdefe14 None admin] RBAC: Adding query filter params (name=object-store) {{(pid=81438) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:54:50.571410 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d38cabd-f247-4879-bb63-c56e6bdefe14 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:50.572770 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d38cabd-f247-4879-bb63-c56e6bdefe14 None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:50.574594 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:54:50.578416 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 456/922] 10.222.0.50 () {56 vars in 1122 bytes} [Tue Sep 13 15:54:50 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:54:50.584048 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43d3f089-7837-4ac9-b761-ab6d36c182c9 None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:50.685682 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:54:50.696321 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43d3f089-7837-4ac9-b761-ab6d36c182c9 None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:50.735938 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc4e5a83-6c84-4ed2-b1c4-210fa847a338 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:50.736596 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc4e5a83-6c84-4ed2-b1c4-210fa847a338 None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:50.736832 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc4e5a83-6c84-4ed2-b1c4-210fa847a338 None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:50.737084 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc4e5a83-6c84-4ed2-b1c4-210fa847a338 None admin] PATH_INFO: `/v3/services` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:50.737776 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc4e5a83-6c84-4ed2-b1c4-210fa847a338 None admin] RBAC: Adding query filter params (type=object-store) {{(pid=81437) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:54:50.738086 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc4e5a83-6c84-4ed2-b1c4-210fa847a338 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:50.739504 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc4e5a83-6c84-4ed2-b1c4-210fa847a338 None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:50.745487 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 467/923] 10.222.0.50 () {56 vars in 1122 bytes} [Tue Sep 13 15:54:50 2022] GET /identity/v3/services?type=object-store => generated 356 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 13 15:54:50.751476 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d38cabd-f247-4879-bb63-c56e6bdefe14 None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:50.810934 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c250b23-09ae-47e8-90a5-ff7906a7223c None None] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:50.811261 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c250b23-09ae-47e8-90a5-ff7906a7223c None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:50.811537 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c250b23-09ae-47e8-90a5-ff7906a7223c None None] PATH_INFO: `/` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:50.813140 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 468/924] 10.222.0.50 () {54 vars in 927 bytes} [Tue Sep 13 15:54:50 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:54:50.818585 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cee866f-fbc5-4691-8b7a-ec3d94642b61 None None] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:50.818874 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cee866f-fbc5-4691-8b7a-ec3d94642b61 None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:50.819178 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cee866f-fbc5-4691-8b7a-ec3d94642b61 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:50.839597 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.auth.core [None req-9cee866f-fbc5-4691-8b7a-ec3d94642b61 None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81437) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:54:50.840807 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-9cee866f-fbc5-4691-8b7a-ec3d94642b61 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:54:50.868783 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d38cabd-f247-4879-bb63-c56e6bdefe14 None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:50.910752 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7630ee6d-c9e3-4a92-83e7-1e78b4ce12f0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:50.911664 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7630ee6d-c9e3-4a92-83e7-1e78b4ce12f0 None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:50.911973 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7630ee6d-c9e3-4a92-83e7-1e78b4ce12f0 None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:50.912642 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7630ee6d-c9e3-4a92-83e7-1e78b4ce12f0 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:50.913660 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7630ee6d-c9e3-4a92-83e7-1e78b4ce12f0 None admin] RBAC: Adding query filter params (interface=public, service_id=f2c069a5bfbd4001a28d284a78e9f24d, region_id=RegionOne) {{(pid=81438) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:54:50.914180 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7630ee6d-c9e3-4a92-83e7-1e78b4ce12f0 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:50.915692 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7630ee6d-c9e3-4a92-83e7-1e78b4ce12f0 None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:50.917361 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 469/925] 10.222.0.50 () {58 vars in 1024 bytes} [Tue Sep 13 15:54:50 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 13 15:54:50.927248 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 457/926] 10.222.0.50 () {56 vars in 1251 bytes} [Tue Sep 13 15:54:50 2022] GET /identity/v3/endpoints?service_id=f2c069a5bfbd4001a28d284a78e9f24d&interface=public®ion_id=RegionOne => generated 194 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:54:50.932811 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cee866f-fbc5-4691-8b7a-ec3d94642b61 None None] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:51.034068 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:54:51.058526 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cee866f-fbc5-4691-8b7a-ec3d94642b61 None None] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:51.100419 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b67941bf-dbb0-4dd2-8214-4081608d7bbc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:51.101034 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b67941bf-dbb0-4dd2-8214-4081608d7bbc None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:51.101407 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b67941bf-dbb0-4dd2-8214-4081608d7bbc None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:51.101687 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b67941bf-dbb0-4dd2-8214-4081608d7bbc None admin] PATH_INFO: `/v3/services` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:51.102292 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b67941bf-dbb0-4dd2-8214-4081608d7bbc None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:51.103691 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b67941bf-dbb0-4dd2-8214-4081608d7bbc None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:51.110110 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 470/927] 10.222.0.50 () {56 vars in 1087 bytes} [Tue Sep 13 15:54:50 2022] GET /identity/v3/services => generated 1497 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 15:54:51.206416 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7630ee6d-c9e3-4a92-83e7-1e78b4ce12f0 None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:51.280645 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:54:51.328453 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7630ee6d-c9e3-4a92-83e7-1e78b4ce12f0 None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:51.362659 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f13ef39d-77ac-4fe4-bad8-49964be17b2a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:51.363326 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f13ef39d-77ac-4fe4-bad8-49964be17b2a None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:51.363670 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f13ef39d-77ac-4fe4-bad8-49964be17b2a None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:51.363947 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f13ef39d-77ac-4fe4-bad8-49964be17b2a None admin] PATH_INFO: `/v3/services/compute_legacy` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:51.364532 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f13ef39d-77ac-4fe4-bad8-49964be17b2a None admin] RBAC: Authorizing `identity:get_service(service_id=compute_legacy)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:51.365907 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f13ef39d-77ac-4fe4-bad8-49964be17b2a None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:51.375869 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application [None req-f13ef39d-77ac-4fe4-bad8-49964be17b2a None admin] Could not find service: compute_legacy.: keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Sep 13 15:54:51.375869 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:54:51.375869 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 13 15:54:51.375869 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 13 15:54:51.375869 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 13 15:54:51.375869 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 13 15:54:51.375869 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 13 15:54:51.375869 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 13 15:54:51.375869 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Sep 13 15:54:51.375869 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Sep 13 15:54:51.375869 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 13 15:54:51.375869 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Sep 13 15:54:51.375869 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:54:51.375869 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:54:51.375869 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:54:51.375869 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:54:51.376852 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:54:51.376852 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:54:51.376852 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:54:51.376852 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:54:51.376852 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:54:51.376852 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:54:51.376852 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:54:51.376852 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 13 15:54:51.376852 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 13 15:54:51.376852 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 13 15:54:51.376852 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 13 15:54:51.376852 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:54:51.376852 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:54:51.376852 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:54:51.376852 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:54:51.376852 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:54:51.376852 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:54:51.377751 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:54:51.377751 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application with Lock( Sep 13 15:54:51.377751 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:54:51.377751 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:54:51.377751 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:54:51.377751 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:54:51.377751 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:54:51.377751 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:54:51.377751 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:54:51.377751 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:54:51.377751 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 13 15:54:51.377751 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 13 15:54:51.377751 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Sep 13 15:54:51.377751 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Sep 13 15:54:51.377751 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 458/928] 10.222.0.50 () {56 vars in 1132 bytes} [Tue Sep 13 15:54:51 2022] GET /identity/v3/services/compute_legacy => generated 95 bytes in 171 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:54:51.384390 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b67941bf-dbb0-4dd2-8214-4081608d7bbc None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:51.399669 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0112a103-6a4c-4195-b315-f2528ca9d897 None None] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:51.399996 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0112a103-6a4c-4195-b315-f2528ca9d897 None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:51.400285 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0112a103-6a4c-4195-b315-f2528ca9d897 None None] PATH_INFO: `/` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:51.401757 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 459/929] 10.222.0.50 () {54 vars in 927 bytes} [Tue Sep 13 15:54:51 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 13 15:54:51.407036 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc221fc7-904b-471f-9cbb-03074e08ece0 None None] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:51.407320 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc221fc7-904b-471f-9cbb-03074e08ece0 None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:51.407596 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc221fc7-904b-471f-9cbb-03074e08ece0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:51.425879 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.auth.core [None req-fc221fc7-904b-471f-9cbb-03074e08ece0 None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81438) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:54:51.427186 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-fc221fc7-904b-471f-9cbb-03074e08ece0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:54:51.503936 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 460/930] 10.222.0.50 () {58 vars in 1024 bytes} [Tue Sep 13 15:54:51 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 13 15:54:51.508188 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b67941bf-dbb0-4dd2-8214-4081608d7bbc None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:51.551708 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11cc283a-c2b9-472e-9951-a93a01a99c05 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:51.552320 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11cc283a-c2b9-472e-9951-a93a01a99c05 None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:51.552602 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11cc283a-c2b9-472e-9951-a93a01a99c05 None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:51.552880 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11cc283a-c2b9-472e-9951-a93a01a99c05 None admin] PATH_INFO: `/v3/services` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:51.553508 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11cc283a-c2b9-472e-9951-a93a01a99c05 None admin] RBAC: Adding query filter params (name=compute_legacy) {{(pid=81437) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:54:51.553816 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11cc283a-c2b9-472e-9951-a93a01a99c05 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:51.555377 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11cc283a-c2b9-472e-9951-a93a01a99c05 None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:51.560827 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 471/931] 10.222.0.50 () {56 vars in 1126 bytes} [Tue Sep 13 15:54:51 2022] GET /identity/v3/services?name=compute_legacy => generated 131 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:54:51.566059 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc221fc7-904b-471f-9cbb-03074e08ece0 None None] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:51.626099 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:54:51.672545 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc221fc7-904b-471f-9cbb-03074e08ece0 None None] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:51.709106 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27ae2a93-96b6-4bb2-8b18-b211670a8516 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:51.710069 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27ae2a93-96b6-4bb2-8b18-b211670a8516 None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:51.711312 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27ae2a93-96b6-4bb2-8b18-b211670a8516 None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:51.711821 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27ae2a93-96b6-4bb2-8b18-b211670a8516 None admin] PATH_INFO: `/v3/services` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:51.712673 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27ae2a93-96b6-4bb2-8b18-b211670a8516 None admin] RBAC: Adding query filter params (type=compute_legacy) {{(pid=81438) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:54:51.713178 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27ae2a93-96b6-4bb2-8b18-b211670a8516 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:51.714753 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27ae2a93-96b6-4bb2-8b18-b211670a8516 None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:51.720716 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 461/932] 10.222.0.50 () {56 vars in 1126 bytes} [Tue Sep 13 15:54:51 2022] GET /identity/v3/services?type=compute_legacy => generated 386 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:54:51.726103 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11cc283a-c2b9-472e-9951-a93a01a99c05 None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:51.743315 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:54:51.793418 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27ae2a93-96b6-4bb2-8b18-b211670a8516 None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:51.844175 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11cc283a-c2b9-472e-9951-a93a01a99c05 None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:51.890300 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e24cf78-5653-454a-b9e4-0c90adb83468 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:51.891078 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e24cf78-5653-454a-b9e4-0c90adb83468 None admin] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:51.891816 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e24cf78-5653-454a-b9e4-0c90adb83468 None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:51.892881 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e24cf78-5653-454a-b9e4-0c90adb83468 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:51.893799 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e24cf78-5653-454a-b9e4-0c90adb83468 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:51.897181 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e24cf78-5653-454a-b9e4-0c90adb83468 None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:51.922684 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27ae2a93-96b6-4bb2-8b18-b211670a8516 None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:51.927061 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 472/933] 10.222.0.50 () {60 vars in 1144 bytes} [Tue Sep 13 15:54:51 2022] POST /identity/v3/endpoints => generated 347 bytes in 201 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 15:54:51.958515 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:54:51.980818 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9c442c4-cdbc-47df-a00b-19a413e17679 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:51.981905 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9c442c4-cdbc-47df-a00b-19a413e17679 None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:51.982393 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9c442c4-cdbc-47df-a00b-19a413e17679 None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:51.982892 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9c442c4-cdbc-47df-a00b-19a413e17679 None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:51.997003 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9c442c4-cdbc-47df-a00b-19a413e17679 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 13 15:54:52.002609 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application [None req-d9c442c4-cdbc-47df-a00b-19a413e17679 None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Sep 13 15:54:52.002609 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:54:52.002609 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:54:52.002609 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:54:52.002609 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:54:52.002609 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:54:52.002609 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:54:52.002609 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:54:52.002609 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:54:52.002609 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:54:52.002609 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:54:52.002609 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:54:52.002609 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 13 15:54:52.002609 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 13 15:54:52.002609 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 13 15:54:52.002609 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 13 15:54:52.003683 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 13 15:54:52.003683 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 13 15:54:52.003683 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 13 15:54:52.003683 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 13 15:54:52.003683 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:54:52.003683 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:54:52.003683 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:54:52.003683 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:54:52.003683 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:54:52.003683 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:54:52.003683 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:54:52.003683 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application with Lock( Sep 13 15:54:52.003683 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:54:52.003683 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:54:52.003683 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:54:52.003683 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:54:52.004529 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:54:52.004529 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:54:52.004529 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:54:52.004529 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:54:52.004529 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 13 15:54:52.004529 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 13 15:54:52.004529 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 13 15:54:52.004529 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 13 15:54:52.004529 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 13 15:54:52.004529 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 13 15:54:52.004529 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Sep 13 15:54:52.004529 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Sep 13 15:54:52.007491 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 462/934] 10.222.0.50 () {56 vars in 1120 bytes} [Tue Sep 13 15:54:51 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 214 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:54:52.016264 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e24cf78-5653-454a-b9e4-0c90adb83468 None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:52.106290 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:54:52.144740 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e24cf78-5653-454a-b9e4-0c90adb83468 None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:52.182762 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-200ed3a5-00f5-43e3-ae43-3e51d5f08828 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:52.183682 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-200ed3a5-00f5-43e3-ae43-3e51d5f08828 None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:52.184114 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-200ed3a5-00f5-43e3-ae43-3e51d5f08828 None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:52.184539 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-200ed3a5-00f5-43e3-ae43-3e51d5f08828 None admin] PATH_INFO: `/v3/roles` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:52.185352 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-200ed3a5-00f5-43e3-ae43-3e51d5f08828 None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=81437) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:54:52.185820 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-200ed3a5-00f5-43e3-ae43-3e51d5f08828 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:52.187395 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-200ed3a5-00f5-43e3-ae43-3e51d5f08828 None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:52.197297 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 473/935] 10.222.0.50 () {56 vars in 1115 bytes} [Tue Sep 13 15:54:52 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:54:52.203015 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9c442c4-cdbc-47df-a00b-19a413e17679 None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:52.315680 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9c442c4-cdbc-47df-a00b-19a413e17679 None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:52.352828 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7bfad3c-22e9-41a2-af21-567cb61381cf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:52.353476 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7bfad3c-22e9-41a2-af21-567cb61381cf None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:52.353755 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7bfad3c-22e9-41a2-af21-567cb61381cf None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:52.354030 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7bfad3c-22e9-41a2-af21-567cb61381cf None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:52.357788 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a88b3765-c968-4825-ad38-ecf5955aec31 None None] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:52.358080 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a88b3765-c968-4825-ad38-ecf5955aec31 None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:52.358358 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a88b3765-c968-4825-ad38-ecf5955aec31 None None] PATH_INFO: `/` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:52.360339 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 474/936] 10.222.0.50 () {54 vars in 927 bytes} [Tue Sep 13 15:54:52 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:54:52.365774 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ccd3719-c5c3-4427-ba2c-9662c1307ea5 None None] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:52.366056 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ccd3719-c5c3-4427-ba2c-9662c1307ea5 None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:52.366332 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ccd3719-c5c3-4427-ba2c-9662c1307ea5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:52.372526 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7bfad3c-22e9-41a2-af21-567cb61381cf None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:52.373968 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7bfad3c-22e9-41a2-af21-567cb61381cf None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:52.389938 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.auth.core [None req-7ccd3719-c5c3-4427-ba2c-9662c1307ea5 None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81437) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:54:52.391166 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-7ccd3719-c5c3-4427-ba2c-9662c1307ea5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:54:52.392296 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application [None req-d7bfad3c-22e9-41a2-af21-567cb61381cf None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:54:52.392296 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:54:52.392296 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:54:52.392296 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:54:52.392296 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:54:52.392296 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:54:52.392296 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:54:52.392296 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:54:52.392296 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:54:52.392296 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:54:52.392296 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:54:52.392296 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:54:52.392296 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 13 15:54:52.392296 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 13 15:54:52.392296 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 13 15:54:52.392296 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 13 15:54:52.393289 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:54:52.393289 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:54:52.393289 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:54:52.393289 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:54:52.393289 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:54:52.393289 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:54:52.393289 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:54:52.393289 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application with Lock( Sep 13 15:54:52.393289 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:54:52.393289 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:54:52.393289 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:54:52.393289 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:54:52.393289 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:54:52.393289 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:54:52.393289 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:54:52.393289 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:54:52.393289 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 13 15:54:52.394235 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 13 15:54:52.394235 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:54:52.394235 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Sep 13 15:54:52.400980 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 463/937] 10.222.0.50 () {56 vars in 1108 bytes} [Tue Sep 13 15:54:52 2022] GET /identity/v3/domains/Default => generated 87 bytes in 193 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 13 15:54:52.402185 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7bfad3c-22e9-41a2-af21-567cb61381cf None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:52.440945 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:54:52.468641 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 475/938] 10.222.0.50 () {58 vars in 1024 bytes} [Tue Sep 13 15:54:52 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 13 15:54:52.524065 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7bfad3c-22e9-41a2-af21-567cb61381cf None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:52.558714 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f915212e-e35a-4d1e-a155-f919cf961f15 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:52.559355 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f915212e-e35a-4d1e-a155-f919cf961f15 None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:52.559673 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f915212e-e35a-4d1e-a155-f919cf961f15 None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:52.559959 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f915212e-e35a-4d1e-a155-f919cf961f15 None admin] PATH_INFO: `/v3/domains` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:52.560600 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f915212e-e35a-4d1e-a155-f919cf961f15 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81438) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:54:52.560955 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f915212e-e35a-4d1e-a155-f919cf961f15 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:52.562325 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f915212e-e35a-4d1e-a155-f919cf961f15 None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:52.574932 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 464/939] 10.222.0.50 () {56 vars in 1109 bytes} [Tue Sep 13 15:54:52 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:54:52.580038 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ccd3719-c5c3-4427-ba2c-9662c1307ea5 None None] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:52.644510 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:54:52.685729 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ccd3719-c5c3-4427-ba2c-9662c1307ea5 None None] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:52.722767 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06c9b1ae-c517-4178-a31f-e1f18789568e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:52.723491 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06c9b1ae-c517-4178-a31f-e1f18789568e None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:52.723825 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06c9b1ae-c517-4178-a31f-e1f18789568e None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:52.724104 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06c9b1ae-c517-4178-a31f-e1f18789568e None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:52.732130 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06c9b1ae-c517-4178-a31f-e1f18789568e None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:52.733644 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06c9b1ae-c517-4178-a31f-e1f18789568e None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:52.746491 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application [None req-06c9b1ae-c517-4178-a31f-e1f18789568e None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Sep 13 15:54:52.746491 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:54:52.746491 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:54:52.746491 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:54:52.746491 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:54:52.746491 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:54:52.746491 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:54:52.746491 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:54:52.746491 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:54:52.746491 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:54:52.746491 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:54:52.746491 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:54:52.746491 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 13 15:54:52.746491 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 13 15:54:52.746491 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 13 15:54:52.746491 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 13 15:54:52.749710 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:54:52.749710 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:54:52.749710 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 13 15:54:52.749710 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 13 15:54:52.749710 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 13 15:54:52.749710 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 13 15:54:52.749710 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:54:52.749710 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:54:52.749710 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:54:52.749710 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:54:52.749710 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:54:52.749710 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application with Lock( Sep 13 15:54:52.749710 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:54:52.749710 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:54:52.749710 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:54:52.749710 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:54:52.749710 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:54:52.749710 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:54:52.750659 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:54:52.750659 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:54:52.750659 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 13 15:54:52.750659 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 13 15:54:52.750659 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 13 15:54:52.750659 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 13 15:54:52.750659 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 13 15:54:52.750659 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 13 15:54:52.750659 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Sep 13 15:54:52.750659 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application Sep 13 15:54:52.750659 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 476/940] 10.222.0.50 () {56 vars in 1117 bytes} [Tue Sep 13 15:54:52 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 169 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 13 15:54:52.758842 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f915212e-e35a-4d1e-a155-f919cf961f15 None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:52.790382 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06c9b1ae-c517-4178-a31f-e1f18789568e None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:52.856840 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:54:52.868598 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f915212e-e35a-4d1e-a155-f919cf961f15 None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:52.903814 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06c9b1ae-c517-4178-a31f-e1f18789568e None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:52.918377 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96b689db-2edb-4709-bfae-5121e7721260 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:52.919154 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96b689db-2edb-4709-bfae-5121e7721260 None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:52.919436 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96b689db-2edb-4709-bfae-5121e7721260 None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:52.919711 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96b689db-2edb-4709-bfae-5121e7721260 None admin] PATH_INFO: `/v3/users` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:52.920499 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96b689db-2edb-4709-bfae-5121e7721260 None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=81438) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:54:52.920811 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96b689db-2edb-4709-bfae-5121e7721260 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:52.922998 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96b689db-2edb-4709-bfae-5121e7721260 None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:52.933183 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:54:52.946993 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 465/941] 10.222.0.50 () {56 vars in 1149 bytes} [Tue Sep 13 15:54:52 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:54:52.949402 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee9302d2-6b69-42a4-8295-1a1e24f49743 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:52.950046 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee9302d2-6b69-42a4-8295-1a1e24f49743 None admin] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:52.950331 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee9302d2-6b69-42a4-8295-1a1e24f49743 None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:52.950605 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee9302d2-6b69-42a4-8295-1a1e24f49743 None admin] PATH_INFO: `/v3/projects` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:52.954961 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee9302d2-6b69-42a4-8295-1a1e24f49743 None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:52.955150 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96b689db-2edb-4709-bfae-5121e7721260 None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:52.955284 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee9302d2-6b69-42a4-8295-1a1e24f49743 None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:52.955627 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: WARNING keystone.server.flask.common [None req-ee9302d2-6b69-42a4-8295-1a1e24f49743 None admin] Deprecated: Not specifying a domain during a create user, group or project call, and relying on falling back to the default domain, is deprecated as of Liberty. There is no plan to remove this compatibility, however, future API versions may remove this, so please specify the domain explicitly or use a domain-scoped token. Sep 13 15:54:52.982647 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.sql.core [None req-ee9302d2-6b69-42a4-8295-1a1e24f49743 None admin] Conflict project: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'default-service' for key 'project.ixu_project_name_domain_id'") Sep 13 15:54:52.982647 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [SQL: INSERT INTO project (id, name, domain_id, description, enabled, extra, parent_id, is_domain) VALUES (%(id)s, %(name)s, %(domain_id)s, %(description)s, %(enabled)s, %(extra)s, %(parent_id)s, %(is_domain)s)] Sep 13 15:54:52.982647 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [parameters: {'id': '4148b09a44f94027b01adb3e151e56fa', 'name': 'service', 'domain_id': 'default', 'description': '', 'enabled': 1, 'extra': '{}', 'parent_id': 'default', 'is_domain': 0}] Sep 13 15:54:52.982647 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: (Background on this error at: https://sqlalche.me/e/14/gkpj) {{(pid=81437) wrapper /opt/stack/keystone/keystone/common/sql/core.py:533}} Sep 13 15:54:52.990059 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application [None req-ee9302d2-6b69-42a4-8295-1a1e24f49743 None admin] Conflict occurred attempting to store project - it is not permitted to have two projects with either the same name or same id in the same domain: name is service, project id 4148b09a44f94027b01adb3e151e56fa.: keystone.exception.Conflict: Conflict occurred attempting to store project - it is not permitted to have two projects with either the same name or same id in the same domain: name is service, project id 4148b09a44f94027b01adb3e151e56fa. Sep 13 15:54:52.990059 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:54:52.990059 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Sep 13 15:54:52.990059 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application self.dialect.do_execute( Sep 13 15:54:52.990059 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Sep 13 15:54:52.990059 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Sep 13 15:54:52.990059 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Sep 13 15:54:52.990059 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application result = self._query(query) Sep 13 15:54:52.990059 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Sep 13 15:54:52.990059 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application conn.query(q) Sep 13 15:54:52.990059 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Sep 13 15:54:52.990059 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Sep 13 15:54:52.990059 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Sep 13 15:54:52.990853 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application result.read() Sep 13 15:54:52.990853 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Sep 13 15:54:52.990853 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Sep 13 15:54:52.990853 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Sep 13 15:54:52.990853 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application packet.raise_for_error() Sep 13 15:54:52.990853 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Sep 13 15:54:52.990853 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Sep 13 15:54:52.990853 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Sep 13 15:54:52.990853 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Sep 13 15:54:52.990853 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application pymysql.err.IntegrityError: (1062, "Duplicate entry 'default-service' for key 'project.ixu_project_name_domain_id'") Sep 13 15:54:52.990853 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application Sep 13 15:54:52.990853 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application The above exception was the direct cause of the following exception: Sep 13 15:54:52.990853 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application Sep 13 15:54:52.990853 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:54:52.990853 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 528, in wrapper Sep 13 15:54:52.990853 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return method(*args, **kwargs) Sep 13 15:54:52.990853 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 241, in create_project Sep 13 15:54:52.990853 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return project_ref.to_dict() Sep 13 15:54:52.992392 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Sep 13 15:54:52.992392 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application next(self.gen) Sep 13 15:54:52.992392 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 1060, in _transaction_scope Sep 13 15:54:52.992392 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application yield resource Sep 13 15:54:52.992392 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Sep 13 15:54:52.992392 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application next(self.gen) Sep 13 15:54:52.992392 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 662, in _session Sep 13 15:54:52.992392 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application self.session.rollback() Sep 13 15:54:52.992392 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Sep 13 15:54:52.992392 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application self.force_reraise() Sep 13 15:54:52.992392 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Sep 13 15:54:52.992392 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application raise self.value Sep 13 15:54:52.992392 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 659, in _session Sep 13 15:54:52.992392 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application self._end_session_transaction(self.session) Sep 13 15:54:52.992392 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 687, in _end_session_transaction Sep 13 15:54:52.992392 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application session.commit() Sep 13 15:54:52.992392 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 1451, in commit Sep 13 15:54:52.993293 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application self._transaction.commit(_to_root=self.future) Sep 13 15:54:52.993293 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 829, in commit Sep 13 15:54:52.993293 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application self._prepare_impl() Sep 13 15:54:52.993293 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 808, in _prepare_impl Sep 13 15:54:52.993293 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application self.session.flush() Sep 13 15:54:52.993293 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3383, in flush Sep 13 15:54:52.993293 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application self._flush(objects) Sep 13 15:54:52.993293 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3523, in _flush Sep 13 15:54:52.993293 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application transaction.rollback(_capture_exception=True) Sep 13 15:54:52.993293 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__ Sep 13 15:54:52.993293 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application compat.raise_( Sep 13 15:54:52.993293 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Sep 13 15:54:52.993293 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application raise exception Sep 13 15:54:52.993293 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3483, in _flush Sep 13 15:54:52.993293 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application flush_context.execute() Sep 13 15:54:52.993293 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 456, in execute Sep 13 15:54:52.993293 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application rec.execute(self) Sep 13 15:54:52.994213 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 630, in execute Sep 13 15:54:52.994213 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application util.preloaded.orm_persistence.save_obj( Sep 13 15:54:52.994213 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 245, in save_obj Sep 13 15:54:52.994213 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application _emit_insert_statements( Sep 13 15:54:52.994213 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 1097, in _emit_insert_statements Sep 13 15:54:52.994213 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application c = connection._execute_20( Sep 13 15:54:52.994213 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1705, in _execute_20 Sep 13 15:54:52.994213 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return meth(self, args_10style, kwargs_10style, execution_options) Sep 13 15:54:52.994213 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/sql/elements.py", line 333, in _execute_on_connection Sep 13 15:54:52.994213 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return connection._execute_clauseelement( Sep 13 15:54:52.994213 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1572, in _execute_clauseelement Sep 13 15:54:52.994213 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application ret = self._execute_context( Sep 13 15:54:52.994213 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1943, in _execute_context Sep 13 15:54:52.994213 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application self._handle_dbapi_exception( Sep 13 15:54:52.994213 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 2122, in _handle_dbapi_exception Sep 13 15:54:52.995038 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application util.raise_(newraise, with_traceback=exc_info[2], from_=e) Sep 13 15:54:52.995038 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Sep 13 15:54:52.995038 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application raise exception Sep 13 15:54:52.995038 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Sep 13 15:54:52.995038 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application self.dialect.do_execute( Sep 13 15:54:52.995038 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Sep 13 15:54:52.995038 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Sep 13 15:54:52.995038 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Sep 13 15:54:52.995038 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application result = self._query(query) Sep 13 15:54:52.995038 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Sep 13 15:54:52.995038 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application conn.query(q) Sep 13 15:54:52.995038 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Sep 13 15:54:52.995038 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Sep 13 15:54:52.995038 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Sep 13 15:54:52.995038 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application result.read() Sep 13 15:54:52.995038 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Sep 13 15:54:52.995038 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Sep 13 15:54:52.997986 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Sep 13 15:54:52.997986 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application packet.raise_for_error() Sep 13 15:54:52.997986 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Sep 13 15:54:52.997986 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Sep 13 15:54:52.997986 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Sep 13 15:54:52.997986 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Sep 13 15:54:52.997986 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application oslo_db.exception.DBDuplicateEntry: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'default-service' for key 'project.ixu_project_name_domain_id'") Sep 13 15:54:52.997986 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application [SQL: INSERT INTO project (id, name, domain_id, description, enabled, extra, parent_id, is_domain) VALUES (%(id)s, %(name)s, %(domain_id)s, %(description)s, %(enabled)s, %(extra)s, %(parent_id)s, %(is_domain)s)] Sep 13 15:54:52.997986 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application [parameters: {'id': '4148b09a44f94027b01adb3e151e56fa', 'name': 'service', 'domain_id': 'default', 'description': '', 'enabled': 1, 'extra': '{}', 'parent_id': 'default', 'is_domain': 0}] Sep 13 15:54:52.997986 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application (Background on this error at: https://sqlalche.me/e/14/gkpj) Sep 13 15:54:52.997986 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application Sep 13 15:54:52.997986 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 13 15:54:52.997986 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application Sep 13 15:54:52.997986 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:54:52.997986 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 216, in create_project Sep 13 15:54:52.999946 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application ret = self.driver.create_project(project_id, project) Sep 13 15:54:52.999946 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 563, in wrapper Sep 13 15:54:52.999946 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application raise exception.Conflict(type=conflict_type, Sep 13 15:54:52.999946 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application keystone.exception.Conflict: Conflict occurred attempting to store project - Duplicate entry found with name service at domain ID default. Sep 13 15:54:52.999946 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application Sep 13 15:54:52.999946 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 13 15:54:52.999946 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application Sep 13 15:54:52.999946 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:54:52.999946 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:54:52.999946 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:54:52.999946 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:54:52.999946 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:54:52.999946 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:54:52.999946 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:54:52.999946 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:54:52.999946 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:54:52.999946 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:54:53.002087 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:54:53.002087 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 176, in post Sep 13 15:54:53.002087 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application ref = PROVIDERS.resource_api.create_project( Sep 13 15:54:53.002087 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:54:53.002087 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:54:53.002087 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 218, in create_project Sep 13 15:54:53.002087 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application raise exception.Conflict( Sep 13 15:54:53.002087 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application keystone.exception.Conflict: Conflict occurred attempting to store project - it is not permitted to have two projects with either the same name or same id in the same domain: name is service, project id 4148b09a44f94027b01adb3e151e56fa. Sep 13 15:54:53.002087 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application Sep 13 15:54:53.005017 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 477/942] 10.222.0.50 () {60 vars in 1140 bytes} [Tue Sep 13 15:54:52 2022] POST /identity/v3/projects => generated 262 bytes in 216 msecs (HTTP/1.1 409) 5 headers in 183 bytes (2 switches on core 0) Sep 13 15:54:53.014960 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee9302d2-6b69-42a4-8295-1a1e24f49743 None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:53.090893 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96b689db-2edb-4709-bfae-5121e7721260 None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:53.129231 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee9302d2-6b69-42a4-8295-1a1e24f49743 None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:53.132854 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26a26cad-eeb1-418c-9662-2b27249d14d3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:53.133692 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26a26cad-eeb1-418c-9662-2b27249d14d3 None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:53.134146 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26a26cad-eeb1-418c-9662-2b27249d14d3 None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:53.134590 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26a26cad-eeb1-418c-9662-2b27249d14d3 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:53.145426 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26a26cad-eeb1-418c-9662-2b27249d14d3 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:53.147380 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26a26cad-eeb1-418c-9662-2b27249d14d3 None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:53.157588 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:54:53.165510 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application [None req-26a26cad-eeb1-418c-9662-2b27249d14d3 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:54:53.165510 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:54:53.165510 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:54:53.165510 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:54:53.165510 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:54:53.165510 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:54:53.165510 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:54:53.165510 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:54:53.165510 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:54:53.165510 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:54:53.165510 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:54:53.165510 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:54:53.165510 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 13 15:54:53.165510 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 13 15:54:53.165510 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 13 15:54:53.165510 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 13 15:54:53.166555 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:54:53.166555 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:54:53.166555 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:54:53.166555 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:54:53.166555 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:54:53.166555 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:54:53.166555 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:54:53.166555 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application with Lock( Sep 13 15:54:53.166555 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:54:53.166555 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:54:53.166555 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:54:53.166555 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:54:53.166555 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:54:53.166555 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:54:53.166555 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:54:53.166555 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:54:53.166555 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 13 15:54:53.167576 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 13 15:54:53.167576 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:54:53.167576 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Sep 13 15:54:53.169763 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 466/943] 10.222.0.50 () {56 vars in 1108 bytes} [Tue Sep 13 15:54:52 2022] GET /identity/v3/domains/Default => generated 87 bytes in 218 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 13 15:54:53.172653 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddd44abb-380c-4a56-8c0d-8c0f4a8ddf66 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:53.173535 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddd44abb-380c-4a56-8c0d-8c0f4a8ddf66 None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:53.174023 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddd44abb-380c-4a56-8c0d-8c0f4a8ddf66 None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:53.174603 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddd44abb-380c-4a56-8c0d-8c0f4a8ddf66 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:53.175414 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26a26cad-eeb1-418c-9662-2b27249d14d3 None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:53.183257 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ddd44abb-380c-4a56-8c0d-8c0f4a8ddf66 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:53.185373 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ddd44abb-380c-4a56-8c0d-8c0f4a8ddf66 None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:53.198745 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application [None req-ddd44abb-380c-4a56-8c0d-8c0f4a8ddf66 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 13 15:54:53.198745 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:54:53.198745 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:54:53.198745 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:54:53.198745 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:54:53.198745 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:54:53.198745 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:54:53.198745 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:54:53.198745 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:54:53.198745 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:54:53.198745 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:54:53.198745 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:54:53.198745 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 13 15:54:53.198745 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 13 15:54:53.198745 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 13 15:54:53.199672 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 13 15:54:53.199672 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:54:53.199672 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:54:53.199672 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:54:53.199672 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:54:53.199672 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:54:53.199672 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:54:53.199672 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:54:53.199672 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application with Lock( Sep 13 15:54:53.199672 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:54:53.199672 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:54:53.199672 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:54:53.199672 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:54:53.199672 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:54:53.199672 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:54:53.199672 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:54:53.199672 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:54:53.200666 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 13 15:54:53.200666 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 13 15:54:53.200666 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 13 15:54:53.200666 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 13 15:54:53.200666 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 13 15:54:53.200666 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 13 15:54:53.200666 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 13 15:54:53.200666 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application Sep 13 15:54:53.207350 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 478/944] 10.222.0.50 () {56 vars in 1111 bytes} [Tue Sep 13 15:54:53 2022] GET /identity/v3/projects/service => generated 88 bytes in 193 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 13 15:54:53.208474 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddd44abb-380c-4a56-8c0d-8c0f4a8ddf66 None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:53.261615 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:54:53.296494 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26a26cad-eeb1-418c-9662-2b27249d14d3 None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:53.335027 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-886bc764-7fb1-4ad1-85c9-8879b7a798bb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:53.336227 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-886bc764-7fb1-4ad1-85c9-8879b7a798bb None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:53.336516 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-886bc764-7fb1-4ad1-85c9-8879b7a798bb None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:53.336792 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-886bc764-7fb1-4ad1-85c9-8879b7a798bb None admin] PATH_INFO: `/v3/domains` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:53.337431 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-886bc764-7fb1-4ad1-85c9-8879b7a798bb None admin] RBAC: Adding query filter params (name=Default) {{(pid=81438) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:54:53.337928 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-886bc764-7fb1-4ad1-85c9-8879b7a798bb None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:53.339134 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-886bc764-7fb1-4ad1-85c9-8879b7a798bb None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:53.356830 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 467/945] 10.222.0.50 () {56 vars in 1109 bytes} [Tue Sep 13 15:54:53 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 13 15:54:53.357520 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddd44abb-380c-4a56-8c0d-8c0f4a8ddf66 None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:53.357754 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-886bc764-7fb1-4ad1-85c9-8879b7a798bb None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:53.396433 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e3b6f44-00c6-4ecc-90ee-9617b3600205 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:53.397037 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e3b6f44-00c6-4ecc-90ee-9617b3600205 None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:53.397310 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e3b6f44-00c6-4ecc-90ee-9617b3600205 None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:53.397577 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e3b6f44-00c6-4ecc-90ee-9617b3600205 None admin] PATH_INFO: `/v3/projects` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:53.398445 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e3b6f44-00c6-4ecc-90ee-9617b3600205 None admin] RBAC: Adding query filter params (name=service) {{(pid=81437) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:54:53.398749 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e3b6f44-00c6-4ecc-90ee-9617b3600205 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:53.400216 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e3b6f44-00c6-4ecc-90ee-9617b3600205 None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:53.426997 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 479/946] 10.222.0.50 () {56 vars in 1112 bytes} [Tue Sep 13 15:54:53 2022] GET /identity/v3/projects?name=service => generated 411 bytes in 213 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:54:53.426997 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae532285-0de2-493d-ade5-7e3da3aff335 None None] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:53.426997 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae532285-0de2-493d-ade5-7e3da3aff335 None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:53.426997 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae532285-0de2-493d-ade5-7e3da3aff335 None None] PATH_INFO: `/` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:53.426997 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 480/947] 10.222.0.50 () {54 vars in 927 bytes} [Tue Sep 13 15:54:53 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:54:53.428634 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:54:53.430005 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd247b6f-5cff-454c-a216-f46d0b68dcbc None None] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:53.430478 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd247b6f-5cff-454c-a216-f46d0b68dcbc None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:53.430931 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd247b6f-5cff-454c-a216-f46d0b68dcbc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:53.451633 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.auth.core [None req-fd247b6f-5cff-454c-a216-f46d0b68dcbc None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81437) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:54:53.453108 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-fd247b6f-5cff-454c-a216-f46d0b68dcbc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:54:53.478358 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-886bc764-7fb1-4ad1-85c9-8879b7a798bb None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:53.517442 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-868530fd-1a24-4563-990a-ad22fd36e7d1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:53.518139 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-868530fd-1a24-4563-990a-ad22fd36e7d1 None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:53.518422 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-868530fd-1a24-4563-990a-ad22fd36e7d1 None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:53.518819 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-868530fd-1a24-4563-990a-ad22fd36e7d1 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:53.527692 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-868530fd-1a24-4563-990a-ad22fd36e7d1 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:53.529157 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-868530fd-1a24-4563-990a-ad22fd36e7d1 None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:53.533004 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 481/948] 10.222.0.50 () {58 vars in 1024 bytes} [Tue Sep 13 15:54:53 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 13 15:54:53.549103 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application [None req-868530fd-1a24-4563-990a-ad22fd36e7d1 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 13 15:54:53.549103 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:54:53.549103 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:54:53.549103 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:54:53.549103 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:54:53.549103 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:54:53.549103 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:54:53.549103 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:54:53.549103 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:54:53.549103 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:54:53.549103 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:54:53.549103 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:54:53.549103 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 13 15:54:53.549103 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 13 15:54:53.549103 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 13 15:54:53.550095 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 13 15:54:53.550095 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:54:53.550095 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:54:53.550095 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:54:53.550095 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:54:53.550095 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:54:53.550095 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:54:53.550095 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:54:53.550095 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application with Lock( Sep 13 15:54:53.550095 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:54:53.550095 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:54:53.550095 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:54:53.550095 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:54:53.550095 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:54:53.550095 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:54:53.550095 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:54:53.550095 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:54:53.551021 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 13 15:54:53.551021 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 13 15:54:53.551021 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 13 15:54:53.551021 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 13 15:54:53.551021 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 13 15:54:53.551021 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 13 15:54:53.551021 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 13 15:54:53.551021 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Sep 13 15:54:53.551021 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 468/949] 10.222.0.50 () {56 vars in 1111 bytes} [Tue Sep 13 15:54:53 2022] GET /identity/v3/projects/service => generated 88 bytes in 193 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:54:53.556929 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd247b6f-5cff-454c-a216-f46d0b68dcbc None None] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:53.592491 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:54:53.675527 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd247b6f-5cff-454c-a216-f46d0b68dcbc None None] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:53.713857 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8aefd18-f5ab-47a6-9c31-f6c351848e6b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:53.714400 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8aefd18-f5ab-47a6-9c31-f6c351848e6b None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:53.714677 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8aefd18-f5ab-47a6-9c31-f6c351848e6b None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:53.714968 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8aefd18-f5ab-47a6-9c31-f6c351848e6b None admin] PATH_INFO: `/v3/projects` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:53.715670 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8aefd18-f5ab-47a6-9c31-f6c351848e6b None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81437) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:54:53.715973 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8aefd18-f5ab-47a6-9c31-f6c351848e6b None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:53.717407 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8aefd18-f5ab-47a6-9c31-f6c351848e6b None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:53.730268 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 482/950] 10.222.0.50 () {56 vars in 1148 bytes} [Tue Sep 13 15:54:53 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:54:53.737499 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-868530fd-1a24-4563-990a-ad22fd36e7d1 None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:53.833589 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8aefd18-f5ab-47a6-9c31-f6c351848e6b None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:53.844743 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-868530fd-1a24-4563-990a-ad22fd36e7d1 None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:53.868573 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:54:53.884498 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e53583ac-cfd8-4fe6-8cbb-c47837c63a08 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:53.885541 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e53583ac-cfd8-4fe6-8cbb-c47837c63a08 None admin] REQUEST_METHOD: `PUT` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:53.885971 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e53583ac-cfd8-4fe6-8cbb-c47837c63a08 None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:53.886402 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e53583ac-cfd8-4fe6-8cbb-c47837c63a08 None admin] PATH_INFO: `/v3/projects/5811e3b83a1e462e9320cad79a3ad791/users/f8364b1a190d4218bf353a89bdb81448/roles/20e2f11234344900a51fc76fe5b625f3` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:53.892730 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e53583ac-cfd8-4fe6-8cbb-c47837c63a08 None admin] RBAC: Authorizing `identity:create_grant(project_id=5811e3b83a1e462e9320cad79a3ad791, user_id=f8364b1a190d4218bf353a89bdb81448, role_id=20e2f11234344900a51fc76fe5b625f3)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:53.894311 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e53583ac-cfd8-4fe6-8cbb-c47837c63a08 None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:53.911991 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 469/951] 10.222.0.50 () {58 vars in 1439 bytes} [Tue Sep 13 15:54:53 2022] PUT /identity/v3/projects/5811e3b83a1e462e9320cad79a3ad791/users/f8364b1a190d4218bf353a89bdb81448/roles/20e2f11234344900a51fc76fe5b625f3 => generated 0 bytes in 175 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:54:53.948728 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:54:53.964535 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8aefd18-f5ab-47a6-9c31-f6c351848e6b None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:54.005299 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35caf263-62cb-4a2f-9327-b5a863ed4a26 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:54.006213 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35caf263-62cb-4a2f-9327-b5a863ed4a26 None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:54.006662 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35caf263-62cb-4a2f-9327-b5a863ed4a26 None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:54.007099 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35caf263-62cb-4a2f-9327-b5a863ed4a26 None admin] PATH_INFO: `/v3/services/object-store` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:54.008206 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35caf263-62cb-4a2f-9327-b5a863ed4a26 None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:54.009881 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35caf263-62cb-4a2f-9327-b5a863ed4a26 None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:54.020157 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application [None req-35caf263-62cb-4a2f-9327-b5a863ed4a26 None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Sep 13 15:54:54.020157 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:54:54.020157 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 13 15:54:54.020157 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 13 15:54:54.020157 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 13 15:54:54.020157 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 13 15:54:54.020157 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 13 15:54:54.020157 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 13 15:54:54.020157 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Sep 13 15:54:54.020157 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application Sep 13 15:54:54.020157 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 13 15:54:54.020157 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application Sep 13 15:54:54.020157 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:54:54.020157 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:54:54.020157 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:54:54.020157 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:54:54.020157 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:54:54.021522 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:54:54.021522 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:54:54.021522 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:54:54.021522 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:54:54.021522 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:54:54.021522 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:54:54.021522 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 13 15:54:54.021522 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 13 15:54:54.021522 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 13 15:54:54.021522 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 13 15:54:54.021522 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:54:54.021522 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:54:54.021522 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:54:54.021522 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:54:54.021522 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:54:54.021522 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:54:54.022580 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:54:54.022580 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application with Lock( Sep 13 15:54:54.022580 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:54:54.022580 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:54:54.022580 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:54:54.022580 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:54:54.022580 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:54:54.022580 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:54:54.022580 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:54:54.022580 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:54:54.022580 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 13 15:54:54.022580 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 13 15:54:54.022580 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Sep 13 15:54:54.022580 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application Sep 13 15:54:54.025668 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 483/952] 10.222.0.50 () {56 vars in 1126 bytes} [Tue Sep 13 15:54:53 2022] GET /identity/v3/services/object-store => generated 93 bytes in 192 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 13 15:54:54.034745 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e53583ac-cfd8-4fe6-8cbb-c47837c63a08 None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:54.147531 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e53583ac-cfd8-4fe6-8cbb-c47837c63a08 None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:54.184039 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f4e7145-2d44-4893-9bab-844d36a7e69d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:54.184704 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f4e7145-2d44-4893-9bab-844d36a7e69d None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:54.184986 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f4e7145-2d44-4893-9bab-844d36a7e69d None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:54.185258 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f4e7145-2d44-4893-9bab-844d36a7e69d None admin] PATH_INFO: `/v3/services` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:54.185885 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f4e7145-2d44-4893-9bab-844d36a7e69d None admin] RBAC: Adding query filter params (name=object-store) {{(pid=81438) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:54:54.186188 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f4e7145-2d44-4893-9bab-844d36a7e69d None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:54.187636 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f4e7145-2d44-4893-9bab-844d36a7e69d None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:54.193085 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 470/953] 10.222.0.50 () {56 vars in 1122 bytes} [Tue Sep 13 15:54:54 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:54:54.199066 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35caf263-62cb-4a2f-9327-b5a863ed4a26 None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:54.308031 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35caf263-62cb-4a2f-9327-b5a863ed4a26 None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:54.323588 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dacbbee6-562c-471d-8edd-ed1ddb13ef34 None None] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:54.324245 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dacbbee6-562c-471d-8edd-ed1ddb13ef34 None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:54.324640 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dacbbee6-562c-471d-8edd-ed1ddb13ef34 None None] PATH_INFO: `/` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:54.327443 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 471/954] 10.222.0.50 () {54 vars in 927 bytes} [Tue Sep 13 15:54:54 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 13 15:54:54.331585 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e392397-5c65-49f6-a873-1afb7923263a None None] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:54.331981 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e392397-5c65-49f6-a873-1afb7923263a None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:54.332689 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e392397-5c65-49f6-a873-1afb7923263a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:54.349267 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0122a997-4fda-4762-8936-e24add0b6ec7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:54.350124 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0122a997-4fda-4762-8936-e24add0b6ec7 None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:54.350548 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0122a997-4fda-4762-8936-e24add0b6ec7 None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:54.351047 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0122a997-4fda-4762-8936-e24add0b6ec7 None admin] PATH_INFO: `/v3/services` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:54.351994 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0122a997-4fda-4762-8936-e24add0b6ec7 None admin] RBAC: Adding query filter params (type=object-store) {{(pid=81437) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:54:54.352440 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0122a997-4fda-4762-8936-e24add0b6ec7 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:54.352770 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.auth.core [None req-9e392397-5c65-49f6-a873-1afb7923263a None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81438) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:54:54.354068 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-9e392397-5c65-49f6-a873-1afb7923263a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:54:54.354308 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0122a997-4fda-4762-8936-e24add0b6ec7 None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:54.361455 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 484/955] 10.222.0.50 () {56 vars in 1122 bytes} [Tue Sep 13 15:54:54 2022] GET /identity/v3/services?type=object-store => generated 356 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:54:54.367631 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0122a997-4fda-4762-8936-e24add0b6ec7 None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:54.392042 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:54:54.402932 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:54:54.438953 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 472/956] 10.222.0.50 () {58 vars in 1024 bytes} [Tue Sep 13 15:54:54 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 13 15:54:54.489973 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0122a997-4fda-4762-8936-e24add0b6ec7 None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:54.525317 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b84cda0a-fcad-4dea-ac8b-d5f7cb109d0a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:54.526327 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b84cda0a-fcad-4dea-ac8b-d5f7cb109d0a None admin] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:54.526839 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b84cda0a-fcad-4dea-ac8b-d5f7cb109d0a None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:54.527331 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b84cda0a-fcad-4dea-ac8b-d5f7cb109d0a None admin] PATH_INFO: `/v3/endpoints` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:54.528232 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b84cda0a-fcad-4dea-ac8b-d5f7cb109d0a None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:54.529830 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b84cda0a-fcad-4dea-ac8b-d5f7cb109d0a None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:54.554282 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 485/957] 10.222.0.50 () {60 vars in 1144 bytes} [Tue Sep 13 15:54:54 2022] POST /identity/v3/endpoints => generated 349 bytes in 188 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 15:54:54.716280 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e392397-5c65-49f6-a873-1afb7923263a None None] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:54.830274 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e392397-5c65-49f6-a873-1afb7923263a None None] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:54.873099 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da281dad-e554-4c59-8ef3-440c8b949244 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:54.873713 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da281dad-e554-4c59-8ef3-440c8b949244 None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:54.873998 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da281dad-e554-4c59-8ef3-440c8b949244 None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:54.874265 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da281dad-e554-4c59-8ef3-440c8b949244 None admin] PATH_INFO: `/v3/services/compute` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:54.874837 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da281dad-e554-4c59-8ef3-440c8b949244 None admin] RBAC: Authorizing `identity:get_service(service_id=compute)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:54.876290 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da281dad-e554-4c59-8ef3-440c8b949244 None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:54.885722 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application [None req-da281dad-e554-4c59-8ef3-440c8b949244 None admin] Could not find service: compute.: keystone.exception.ServiceNotFound: Could not find service: compute. Sep 13 15:54:54.885722 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:54:54.885722 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 13 15:54:54.885722 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 13 15:54:54.885722 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 13 15:54:54.885722 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 13 15:54:54.885722 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 13 15:54:54.885722 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 13 15:54:54.885722 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Sep 13 15:54:54.885722 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Sep 13 15:54:54.885722 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 13 15:54:54.885722 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Sep 13 15:54:54.885722 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:54:54.885722 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:54:54.885722 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:54:54.885722 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:54:54.885722 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:54:54.886765 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:54:54.886765 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:54:54.886765 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:54:54.886765 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:54:54.886765 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:54:54.886765 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:54:54.886765 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 13 15:54:54.886765 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 13 15:54:54.886765 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 13 15:54:54.886765 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 13 15:54:54.886765 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:54:54.886765 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:54:54.886765 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:54:54.886765 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:54:54.886765 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:54:54.886765 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:54:54.888035 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:54:54.888035 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application with Lock( Sep 13 15:54:54.888035 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:54:54.888035 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:54:54.888035 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:54:54.888035 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:54:54.888035 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:54:54.888035 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:54:54.888035 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:54:54.888035 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:54:54.888035 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 13 15:54:54.888035 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 13 15:54:54.888035 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Sep 13 15:54:54.888035 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Sep 13 15:54:54.888035 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 473/958] 10.222.0.50 () {56 vars in 1111 bytes} [Tue Sep 13 15:54:54 2022] GET /identity/v3/services/compute => generated 88 bytes in 171 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:54:54.895035 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b84cda0a-fcad-4dea-ac8b-d5f7cb109d0a None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:54.982179 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:54:55.009936 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b84cda0a-fcad-4dea-ac8b-d5f7cb109d0a None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:55.045497 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06857755-9ff1-4677-ba11-fea012031129 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:55.045972 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06857755-9ff1-4677-ba11-fea012031129 None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:55.046245 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06857755-9ff1-4677-ba11-fea012031129 None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:55.046518 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06857755-9ff1-4677-ba11-fea012031129 None admin] PATH_INFO: `/v3/services` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:55.047176 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06857755-9ff1-4677-ba11-fea012031129 None admin] RBAC: Adding query filter params (name=compute) {{(pid=81437) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:54:55.047477 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06857755-9ff1-4677-ba11-fea012031129 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:55.048900 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06857755-9ff1-4677-ba11-fea012031129 None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:55.054779 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 486/959] 10.222.0.50 () {56 vars in 1112 bytes} [Tue Sep 13 15:54:54 2022] GET /identity/v3/services?name=compute => generated 124 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:54:55.066349 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da281dad-e554-4c59-8ef3-440c8b949244 None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:55.103081 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:54:55.173317 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da281dad-e554-4c59-8ef3-440c8b949244 None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:55.213740 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca57aeec-ec29-4c6e-b757-51260b4f1323 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:55.214348 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca57aeec-ec29-4c6e-b757-51260b4f1323 None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:55.214629 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca57aeec-ec29-4c6e-b757-51260b4f1323 None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:55.214915 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca57aeec-ec29-4c6e-b757-51260b4f1323 None admin] PATH_INFO: `/v3/services` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:55.215561 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca57aeec-ec29-4c6e-b757-51260b4f1323 None admin] RBAC: Adding query filter params (type=compute) {{(pid=81438) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:54:55.215863 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca57aeec-ec29-4c6e-b757-51260b4f1323 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:55.217221 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca57aeec-ec29-4c6e-b757-51260b4f1323 None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:55.222230 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 474/960] 10.222.0.50 () {56 vars in 1112 bytes} [Tue Sep 13 15:54:55 2022] GET /identity/v3/services?type=compute => generated 124 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:54:55.875187 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32d282d0-0e53-4860-8628-35f9e68de00b None None] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:55.875572 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32d282d0-0e53-4860-8628-35f9e68de00b None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:55.875756 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32d282d0-0e53-4860-8628-35f9e68de00b None None] PATH_INFO: `/` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:55.877311 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 487/961] 10.222.0.50 () {54 vars in 927 bytes} [Tue Sep 13 15:54:55 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:54:55.883048 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f33d1b7-ea1b-4a35-b3da-bdc3432d44cd None None] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:55.883519 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f33d1b7-ea1b-4a35-b3da-bdc3432d44cd None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:55.883658 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f33d1b7-ea1b-4a35-b3da-bdc3432d44cd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:55.902798 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.auth.core [None req-3f33d1b7-ea1b-4a35-b3da-bdc3432d44cd None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81438) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:54:55.904014 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-3f33d1b7-ea1b-4a35-b3da-bdc3432d44cd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:54:55.977798 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 475/962] 10.222.0.50 () {58 vars in 1024 bytes} [Tue Sep 13 15:54:55 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 13 15:54:56.246052 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d49811e-8fc1-44e1-81fc-e5cb00c2a84b None None] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:56.246547 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d49811e-8fc1-44e1-81fc-e5cb00c2a84b None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:56.246547 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d49811e-8fc1-44e1-81fc-e5cb00c2a84b None None] PATH_INFO: `/` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:56.248206 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 488/963] 10.222.0.50 () {54 vars in 927 bytes} [Tue Sep 13 15:54:56 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:54:56.253353 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebee8956-8dae-4edb-9adf-30f1e31601bb None None] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:56.253643 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebee8956-8dae-4edb-9adf-30f1e31601bb None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:56.253920 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebee8956-8dae-4edb-9adf-30f1e31601bb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:56.258836 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d49811e-8fc1-44e1-81fc-e5cb00c2a84b None None] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:56.274248 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.auth.core [None req-ebee8956-8dae-4edb-9adf-30f1e31601bb None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81438) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:54:56.275456 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-ebee8956-8dae-4edb-9adf-30f1e31601bb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:54:56.303664 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:54:56.370351 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 476/964] 10.222.0.50 () {58 vars in 1024 bytes} [Tue Sep 13 15:54:56 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 118 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 13 15:54:56.382785 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d49811e-8fc1-44e1-81fc-e5cb00c2a84b None None] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:56.419459 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f61948b-0ec3-4a41-9fe9-435d0819d65a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:56.420449 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f61948b-0ec3-4a41-9fe9-435d0819d65a None admin] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:56.420949 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f61948b-0ec3-4a41-9fe9-435d0819d65a None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:56.421421 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f61948b-0ec3-4a41-9fe9-435d0819d65a None admin] PATH_INFO: `/v3/users` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:56.422267 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f61948b-0ec3-4a41-9fe9-435d0819d65a None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:56.423934 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f61948b-0ec3-4a41-9fe9-435d0819d65a None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:56.435233 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:54:56.442945 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.sql.core [None req-4f61948b-0ec3-4a41-9fe9-435d0819d65a None admin] Conflict user: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'default-cinder' for key 'local_user.domain_id'") Sep 13 15:54:56.442945 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [SQL: INSERT INTO local_user (user_id, domain_id, name, failed_auth_count, failed_auth_at) VALUES (%(user_id)s, %(domain_id)s, %(name)s, %(failed_auth_count)s, %(failed_auth_at)s)] Sep 13 15:54:56.442945 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [parameters: {'user_id': '91f1cf480754454e9ed88467022feb0f', 'domain_id': 'default', 'name': 'cinder', 'failed_auth_count': 0, 'failed_auth_at': None}] Sep 13 15:54:56.442945 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: (Background on this error at: https://sqlalche.me/e/14/gkpj) {{(pid=81437) wrapper /opt/stack/keystone/keystone/common/sql/core.py:533}} Sep 13 15:54:56.452186 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application [None req-4f61948b-0ec3-4a41-9fe9-435d0819d65a None admin] Conflict occurred attempting to store user - Duplicate entry found with name cinder at domain ID default.: keystone.exception.Conflict: Conflict occurred attempting to store user - Duplicate entry found with name cinder at domain ID default. Sep 13 15:54:56.452186 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:54:56.452186 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Sep 13 15:54:56.452186 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application self.dialect.do_execute( Sep 13 15:54:56.452186 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Sep 13 15:54:56.452186 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Sep 13 15:54:56.452186 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Sep 13 15:54:56.452186 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application result = self._query(query) Sep 13 15:54:56.452186 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Sep 13 15:54:56.452186 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application conn.query(q) Sep 13 15:54:56.452186 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Sep 13 15:54:56.452186 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Sep 13 15:54:56.452186 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Sep 13 15:54:56.452186 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application result.read() Sep 13 15:54:56.452186 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Sep 13 15:54:56.453081 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Sep 13 15:54:56.453081 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Sep 13 15:54:56.453081 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application packet.raise_for_error() Sep 13 15:54:56.453081 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Sep 13 15:54:56.453081 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Sep 13 15:54:56.453081 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Sep 13 15:54:56.453081 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Sep 13 15:54:56.453081 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application pymysql.err.IntegrityError: (1062, "Duplicate entry 'default-cinder' for key 'local_user.domain_id'") Sep 13 15:54:56.453081 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application Sep 13 15:54:56.453081 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application The above exception was the direct cause of the following exception: Sep 13 15:54:56.453081 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application Sep 13 15:54:56.453081 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:54:56.453081 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 528, in wrapper Sep 13 15:54:56.453081 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return method(*args, **kwargs) Sep 13 15:54:56.453081 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 141, in create_user Sep 13 15:54:56.453081 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return base.filter_user(user_ref.to_dict()) Sep 13 15:54:56.453081 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Sep 13 15:54:56.453081 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application next(self.gen) Sep 13 15:54:56.454117 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 1060, in _transaction_scope Sep 13 15:54:56.454117 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application yield resource Sep 13 15:54:56.454117 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Sep 13 15:54:56.454117 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application next(self.gen) Sep 13 15:54:56.454117 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 662, in _session Sep 13 15:54:56.454117 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application self.session.rollback() Sep 13 15:54:56.454117 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Sep 13 15:54:56.454117 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application self.force_reraise() Sep 13 15:54:56.454117 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Sep 13 15:54:56.454117 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application raise self.value Sep 13 15:54:56.454117 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 659, in _session Sep 13 15:54:56.454117 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application self._end_session_transaction(self.session) Sep 13 15:54:56.454117 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 687, in _end_session_transaction Sep 13 15:54:56.454117 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application session.commit() Sep 13 15:54:56.454117 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 1451, in commit Sep 13 15:54:56.454117 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application self._transaction.commit(_to_root=self.future) Sep 13 15:54:56.454117 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 829, in commit Sep 13 15:54:56.455068 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application self._prepare_impl() Sep 13 15:54:56.455068 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 808, in _prepare_impl Sep 13 15:54:56.455068 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application self.session.flush() Sep 13 15:54:56.455068 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3383, in flush Sep 13 15:54:56.455068 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application self._flush(objects) Sep 13 15:54:56.455068 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3523, in _flush Sep 13 15:54:56.455068 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application transaction.rollback(_capture_exception=True) Sep 13 15:54:56.455068 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__ Sep 13 15:54:56.455068 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application compat.raise_( Sep 13 15:54:56.455068 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Sep 13 15:54:56.455068 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application raise exception Sep 13 15:54:56.455068 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3483, in _flush Sep 13 15:54:56.455068 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application flush_context.execute() Sep 13 15:54:56.455068 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 456, in execute Sep 13 15:54:56.455068 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application rec.execute(self) Sep 13 15:54:56.455068 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 630, in execute Sep 13 15:54:56.455068 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application util.preloaded.orm_persistence.save_obj( Sep 13 15:54:56.455962 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 245, in save_obj Sep 13 15:54:56.455962 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application _emit_insert_statements( Sep 13 15:54:56.455962 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 1238, in _emit_insert_statements Sep 13 15:54:56.455962 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application result = connection._execute_20( Sep 13 15:54:56.455962 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1705, in _execute_20 Sep 13 15:54:56.455962 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return meth(self, args_10style, kwargs_10style, execution_options) Sep 13 15:54:56.455962 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/sql/elements.py", line 333, in _execute_on_connection Sep 13 15:54:56.455962 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return connection._execute_clauseelement( Sep 13 15:54:56.455962 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1572, in _execute_clauseelement Sep 13 15:54:56.455962 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application ret = self._execute_context( Sep 13 15:54:56.455962 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1943, in _execute_context Sep 13 15:54:56.455962 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application self._handle_dbapi_exception( Sep 13 15:54:56.455962 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 2122, in _handle_dbapi_exception Sep 13 15:54:56.455962 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application util.raise_(newraise, with_traceback=exc_info[2], from_=e) Sep 13 15:54:56.455962 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Sep 13 15:54:56.455962 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application raise exception Sep 13 15:54:56.456841 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Sep 13 15:54:56.456841 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application self.dialect.do_execute( Sep 13 15:54:56.456841 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Sep 13 15:54:56.456841 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Sep 13 15:54:56.456841 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Sep 13 15:54:56.456841 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application result = self._query(query) Sep 13 15:54:56.456841 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Sep 13 15:54:56.456841 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application conn.query(q) Sep 13 15:54:56.456841 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Sep 13 15:54:56.456841 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Sep 13 15:54:56.456841 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Sep 13 15:54:56.456841 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application result.read() Sep 13 15:54:56.456841 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Sep 13 15:54:56.456841 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Sep 13 15:54:56.456841 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Sep 13 15:54:56.456841 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application packet.raise_for_error() Sep 13 15:54:56.457718 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Sep 13 15:54:56.457718 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Sep 13 15:54:56.457718 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Sep 13 15:54:56.457718 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Sep 13 15:54:56.457718 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application oslo_db.exception.DBDuplicateEntry: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'default-cinder' for key 'local_user.domain_id'") Sep 13 15:54:56.457718 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application [SQL: INSERT INTO local_user (user_id, domain_id, name, failed_auth_count, failed_auth_at) VALUES (%(user_id)s, %(domain_id)s, %(name)s, %(failed_auth_count)s, %(failed_auth_at)s)] Sep 13 15:54:56.457718 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application [parameters: {'user_id': '91f1cf480754454e9ed88467022feb0f', 'domain_id': 'default', 'name': 'cinder', 'failed_auth_count': 0, 'failed_auth_at': None}] Sep 13 15:54:56.457718 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application (Background on this error at: https://sqlalche.me/e/14/gkpj) Sep 13 15:54:56.457718 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application Sep 13 15:54:56.457718 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 13 15:54:56.457718 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application Sep 13 15:54:56.457718 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:54:56.457718 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:54:56.457718 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:54:56.457718 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:54:56.460069 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:54:56.460069 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:54:56.460069 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:54:56.460069 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:54:56.460069 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:54:56.460069 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:54:56.460069 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:54:56.460069 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 244, in post Sep 13 15:54:56.460069 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.create_user( Sep 13 15:54:56.460069 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:54:56.460069 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:54:56.460069 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 13 15:54:56.460069 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 13 15:54:56.460069 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 13 15:54:56.460069 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 13 15:54:56.460069 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1000, in create_user Sep 13 15:54:56.460069 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application ref = self._create_user_with_federated_objects(user, driver) Sep 13 15:54:56.462839 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 966, in _create_user_with_federated_objects Sep 13 15:54:56.462839 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application user = driver.create_user(user['id'], user) Sep 13 15:54:56.462839 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 563, in wrapper Sep 13 15:54:56.462839 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application raise exception.Conflict(type=conflict_type, Sep 13 15:54:56.462839 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application keystone.exception.Conflict: Conflict occurred attempting to store user - Duplicate entry found with name cinder at domain ID default. Sep 13 15:54:56.462839 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application Sep 13 15:54:56.465298 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 489/965] 10.222.0.50 () {60 vars in 1131 bytes} [Tue Sep 13 15:54:56 2022] POST /identity/v3/users => generated 160 bytes in 207 msecs (HTTP/1.1 409) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:54:56.472017 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebee8956-8dae-4edb-9adf-30f1e31601bb None None] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:56.576767 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebee8956-8dae-4edb-9adf-30f1e31601bb None None] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:56.612074 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb505784-a837-4b08-966c-e67469273212 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:56.612878 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb505784-a837-4b08-966c-e67469273212 None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:56.613160 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb505784-a837-4b08-966c-e67469273212 None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:56.613431 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb505784-a837-4b08-966c-e67469273212 None admin] PATH_INFO: `/v3/users/cinder` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:56.621297 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb505784-a837-4b08-966c-e67469273212 None admin] RBAC: Authorizing `identity:get_user(user_id=cinder)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:56.622738 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb505784-a837-4b08-966c-e67469273212 None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:56.634806 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application [None req-bb505784-a837-4b08-966c-e67469273212 None admin] Could not find user: cinder.: keystone.exception.UserNotFound: Could not find user: cinder. Sep 13 15:54:56.634806 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:54:56.634806 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:54:56.634806 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:54:56.634806 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:54:56.634806 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:54:56.634806 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:54:56.634806 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:54:56.634806 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:54:56.634806 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:54:56.634806 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:54:56.634806 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:54:56.634806 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 13 15:54:56.634806 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 13 15:54:56.634806 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 13 15:54:56.634806 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 13 15:54:56.635852 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:54:56.635852 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:54:56.635852 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 13 15:54:56.635852 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 13 15:54:56.635852 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 13 15:54:56.635852 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 13 15:54:56.635852 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:54:56.635852 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:54:56.635852 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:54:56.635852 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:54:56.635852 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:54:56.635852 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application with Lock( Sep 13 15:54:56.635852 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:54:56.635852 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:54:56.635852 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:54:56.635852 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:54:56.635852 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:54:56.635852 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:54:56.636827 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:54:56.636827 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:54:56.636827 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 13 15:54:56.636827 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 13 15:54:56.636827 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 13 15:54:56.636827 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 13 15:54:56.636827 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 13 15:54:56.636827 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 13 15:54:56.636827 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: cinder. Sep 13 15:54:56.636827 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Sep 13 15:54:56.636827 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 477/966] 10.222.0.50 () {56 vars in 1099 bytes} [Tue Sep 13 15:54:56 2022] GET /identity/v3/users/cinder => generated 84 bytes in 165 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:54:56.641860 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f61948b-0ec3-4a41-9fe9-435d0819d65a None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:56.673699 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb505784-a837-4b08-966c-e67469273212 None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:56.695915 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:54:56.764806 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f61948b-0ec3-4a41-9fe9-435d0819d65a None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:56.804518 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb505784-a837-4b08-966c-e67469273212 None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:56.818529 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-665fc8d7-b96e-4f61-a5fd-f729b2987ebf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:56.819478 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-665fc8d7-b96e-4f61-a5fd-f729b2987ebf None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:56.820739 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-665fc8d7-b96e-4f61-a5fd-f729b2987ebf None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:56.821390 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-665fc8d7-b96e-4f61-a5fd-f729b2987ebf None admin] PATH_INFO: `/v3/users` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:56.822825 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-665fc8d7-b96e-4f61-a5fd-f729b2987ebf None admin] RBAC: Adding query filter params (name=cinder) {{(pid=81437) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:54:56.823424 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-665fc8d7-b96e-4f61-a5fd-f729b2987ebf None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:56.825662 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-665fc8d7-b96e-4f61-a5fd-f729b2987ebf None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:56.847109 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eabc502f-512d-45d5-ad34-256ff5fd6d3d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:56.848233 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eabc502f-512d-45d5-ad34-256ff5fd6d3d None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:56.848679 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eabc502f-512d-45d5-ad34-256ff5fd6d3d None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:56.849095 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eabc502f-512d-45d5-ad34-256ff5fd6d3d None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:56.861283 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eabc502f-512d-45d5-ad34-256ff5fd6d3d None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 13 15:54:56.868052 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application [None req-eabc502f-512d-45d5-ad34-256ff5fd6d3d None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Sep 13 15:54:56.868052 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:54:56.868052 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:54:56.868052 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:54:56.868052 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:54:56.868052 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:54:56.868052 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:54:56.868052 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:54:56.868052 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:54:56.868052 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:54:56.868052 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:54:56.868052 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:54:56.868052 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 13 15:54:56.868052 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 13 15:54:56.868052 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 13 15:54:56.868052 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 13 15:54:56.869227 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 13 15:54:56.869227 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 13 15:54:56.869227 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 13 15:54:56.869227 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 13 15:54:56.869227 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:54:56.869227 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:54:56.869227 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:54:56.869227 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:54:56.869227 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:54:56.869227 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:54:56.869227 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:54:56.869227 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application with Lock( Sep 13 15:54:56.869227 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:54:56.869227 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:54:56.869227 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:54:56.869227 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:54:56.870262 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:54:56.870262 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:54:56.870262 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:54:56.870262 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:54:56.870262 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 13 15:54:56.870262 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 13 15:54:56.870262 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 13 15:54:56.870262 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 13 15:54:56.870262 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 13 15:54:56.870262 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 13 15:54:56.870262 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Sep 13 15:54:56.870262 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Sep 13 15:54:56.872727 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 478/967] 10.222.0.50 () {56 vars in 1120 bytes} [Tue Sep 13 15:54:56 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 199 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 13 15:54:56.882938 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 490/968] 10.222.0.50 () {56 vars in 1101 bytes} [Tue Sep 13 15:54:56 2022] GET /identity/v3/users?name=cinder => generated 354 bytes in 236 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:54:56.883198 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eabc502f-512d-45d5-ad34-256ff5fd6d3d None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:56.899278 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4567d114-c8e6-4e1f-9432-94cb04e4bf53 None None] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:56.899688 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4567d114-c8e6-4e1f-9432-94cb04e4bf53 None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:56.899843 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4567d114-c8e6-4e1f-9432-94cb04e4bf53 None None] PATH_INFO: `/` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:56.903004 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 491/969] 10.222.0.50 () {54 vars in 927 bytes} [Tue Sep 13 15:54:56 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 13 15:54:56.907372 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d84f2cff-b2db-4ac5-9672-69d58f8e6288 None None] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:56.907656 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d84f2cff-b2db-4ac5-9672-69d58f8e6288 None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:56.907935 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d84f2cff-b2db-4ac5-9672-69d58f8e6288 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:56.928868 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.auth.core [None req-d84f2cff-b2db-4ac5-9672-69d58f8e6288 None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81437) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:54:56.931157 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-d84f2cff-b2db-4ac5-9672-69d58f8e6288 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:54:56.983048 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:54:57.003144 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:INSERT=1,keystone:SELECT=99 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:54:57.011746 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eabc502f-512d-45d5-ad34-256ff5fd6d3d None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:57.030963 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 492/970] 10.222.0.50 () {58 vars in 1024 bytes} [Tue Sep 13 15:54:56 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 120 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 13 15:54:57.052236 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b383d5c3-d94d-4c4a-97cf-3bbf75948823 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:57.052857 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b383d5c3-d94d-4c4a-97cf-3bbf75948823 None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:57.053131 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b383d5c3-d94d-4c4a-97cf-3bbf75948823 None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:57.053399 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b383d5c3-d94d-4c4a-97cf-3bbf75948823 None admin] PATH_INFO: `/v3/roles` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:57.054653 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b383d5c3-d94d-4c4a-97cf-3bbf75948823 None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=81438) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:54:57.055176 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b383d5c3-d94d-4c4a-97cf-3bbf75948823 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:57.056596 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b383d5c3-d94d-4c4a-97cf-3bbf75948823 None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:57.066496 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 479/971] 10.222.0.50 () {56 vars in 1115 bytes} [Tue Sep 13 15:54:56 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 188 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 13 15:54:57.072587 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d84f2cff-b2db-4ac5-9672-69d58f8e6288 None None] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:57.185014 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d84f2cff-b2db-4ac5-9672-69d58f8e6288 None None] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:57.223192 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf550c44-2552-4bfd-a612-da9648f13500 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:57.223882 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf550c44-2552-4bfd-a612-da9648f13500 None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:57.224161 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf550c44-2552-4bfd-a612-da9648f13500 None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:57.224432 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf550c44-2552-4bfd-a612-da9648f13500 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:57.236583 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf550c44-2552-4bfd-a612-da9648f13500 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:57.238029 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf550c44-2552-4bfd-a612-da9648f13500 None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:57.253729 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application [None req-cf550c44-2552-4bfd-a612-da9648f13500 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:54:57.253729 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:54:57.253729 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:54:57.253729 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:54:57.253729 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:54:57.253729 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:54:57.253729 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:54:57.253729 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:54:57.253729 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:54:57.253729 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:54:57.253729 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:54:57.253729 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:54:57.253729 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 13 15:54:57.253729 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 13 15:54:57.253729 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 13 15:54:57.253729 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 13 15:54:57.255144 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:54:57.255144 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:54:57.255144 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:54:57.255144 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:54:57.255144 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:54:57.255144 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:54:57.255144 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:54:57.255144 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application with Lock( Sep 13 15:54:57.255144 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:54:57.255144 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:54:57.255144 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:54:57.255144 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:54:57.255144 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:54:57.255144 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:54:57.255144 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:54:57.255144 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:54:57.255144 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 13 15:54:57.256476 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 13 15:54:57.256476 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:54:57.256476 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application Sep 13 15:54:57.256476 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 493/972] 10.222.0.50 () {56 vars in 1108 bytes} [Tue Sep 13 15:54:57 2022] GET /identity/v3/domains/Default => generated 87 bytes in 183 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:54:57.260819 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b383d5c3-d94d-4c4a-97cf-3bbf75948823 None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:57.311193 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf550c44-2552-4bfd-a612-da9648f13500 None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:57.355608 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:54:57.378166 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b383d5c3-d94d-4c4a-97cf-3bbf75948823 None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:57.415520 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ecb895b-17d3-4560-b3eb-59841b2e560b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:57.416308 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ecb895b-17d3-4560-b3eb-59841b2e560b None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:57.416592 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ecb895b-17d3-4560-b3eb-59841b2e560b None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:57.416865 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ecb895b-17d3-4560-b3eb-59841b2e560b None admin] PATH_INFO: `/v3/domains` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:57.417500 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ecb895b-17d3-4560-b3eb-59841b2e560b None admin] RBAC: Adding query filter params (name=Default) {{(pid=81438) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:54:57.417800 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ecb895b-17d3-4560-b3eb-59841b2e560b None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:57.419434 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ecb895b-17d3-4560-b3eb-59841b2e560b None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:57.424521 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf550c44-2552-4bfd-a612-da9648f13500 None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:57.428827 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:54:57.432830 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 480/973] 10.222.0.50 () {56 vars in 1109 bytes} [Tue Sep 13 15:54:57 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:54:57.437958 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ecb895b-17d3-4560-b3eb-59841b2e560b None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:57.464172 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97168adb-4fbd-48cd-8e53-fb956018db77 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:57.464814 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97168adb-4fbd-48cd-8e53-fb956018db77 None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:57.465093 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97168adb-4fbd-48cd-8e53-fb956018db77 None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:57.465363 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97168adb-4fbd-48cd-8e53-fb956018db77 None admin] PATH_INFO: `/v3/services/object-store` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:57.465938 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97168adb-4fbd-48cd-8e53-fb956018db77 None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:57.467355 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97168adb-4fbd-48cd-8e53-fb956018db77 None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:57.476994 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application [None req-97168adb-4fbd-48cd-8e53-fb956018db77 None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Sep 13 15:54:57.476994 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:54:57.476994 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 13 15:54:57.476994 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 13 15:54:57.476994 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 13 15:54:57.476994 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 13 15:54:57.476994 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 13 15:54:57.476994 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 13 15:54:57.476994 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Sep 13 15:54:57.476994 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application Sep 13 15:54:57.476994 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 13 15:54:57.476994 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application Sep 13 15:54:57.476994 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:54:57.476994 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:54:57.476994 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:54:57.476994 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:54:57.476994 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:54:57.478008 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:54:57.478008 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:54:57.478008 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:54:57.478008 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:54:57.478008 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:54:57.478008 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:54:57.478008 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 13 15:54:57.478008 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 13 15:54:57.478008 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 13 15:54:57.478008 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 13 15:54:57.478008 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:54:57.478008 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:54:57.478008 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:54:57.478008 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:54:57.478008 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:54:57.478008 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:54:57.478876 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:54:57.478876 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application with Lock( Sep 13 15:54:57.478876 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:54:57.478876 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:54:57.478876 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:54:57.478876 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:54:57.478876 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:54:57.478876 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:54:57.478876 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:54:57.478876 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:54:57.478876 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 13 15:54:57.478876 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 13 15:54:57.478876 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Sep 13 15:54:57.478876 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application Sep 13 15:54:57.478876 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 494/974] 10.222.0.50 () {56 vars in 1126 bytes} [Tue Sep 13 15:54:57 2022] GET /identity/v3/services/object-store => generated 93 bytes in 167 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 13 15:54:57.480115 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f3fe75a-f8dd-4399-8fbf-5cc9335e299e None None] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:57.480115 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f3fe75a-f8dd-4399-8fbf-5cc9335e299e None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:57.480115 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f3fe75a-f8dd-4399-8fbf-5cc9335e299e None None] PATH_INFO: `/` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:57.481157 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 495/975] 10.222.0.50 () {54 vars in 927 bytes} [Tue Sep 13 15:54:57 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:54:57.483721 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f3fe75a-f8dd-4399-8fbf-5cc9335e299e None None] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:57.542639 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ecb895b-17d3-4560-b3eb-59841b2e560b None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:57.577000 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f544a21a-67a7-4f3b-8ddf-2ca1c6526f3e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:57.577623 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f544a21a-67a7-4f3b-8ddf-2ca1c6526f3e None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:57.577900 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f544a21a-67a7-4f3b-8ddf-2ca1c6526f3e None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:57.578177 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f544a21a-67a7-4f3b-8ddf-2ca1c6526f3e None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:57.584505 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f3fe75a-f8dd-4399-8fbf-5cc9335e299e None None] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:57.585271 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f544a21a-67a7-4f3b-8ddf-2ca1c6526f3e None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:57.586051 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:54:57.586861 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f544a21a-67a7-4f3b-8ddf-2ca1c6526f3e None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:57.598746 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application [None req-f544a21a-67a7-4f3b-8ddf-2ca1c6526f3e None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Sep 13 15:54:57.598746 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:54:57.598746 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:54:57.598746 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:54:57.598746 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:54:57.598746 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:54:57.598746 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:54:57.598746 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:54:57.598746 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:54:57.598746 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:54:57.598746 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:54:57.598746 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:54:57.598746 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 13 15:54:57.598746 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 13 15:54:57.598746 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 13 15:54:57.598746 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 13 15:54:57.599870 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:54:57.599870 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:54:57.599870 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 13 15:54:57.599870 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 13 15:54:57.599870 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 13 15:54:57.599870 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 13 15:54:57.599870 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:54:57.599870 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:54:57.599870 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:54:57.599870 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:54:57.599870 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:54:57.599870 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application with Lock( Sep 13 15:54:57.599870 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:54:57.599870 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:54:57.599870 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:54:57.599870 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:54:57.599870 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:54:57.599870 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:54:57.601451 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:54:57.601451 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:54:57.601451 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 13 15:54:57.601451 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 13 15:54:57.601451 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 13 15:54:57.601451 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 13 15:54:57.601451 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 13 15:54:57.601451 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 13 15:54:57.601451 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Sep 13 15:54:57.601451 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Sep 13 15:54:57.601451 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 481/976] 10.222.0.50 () {56 vars in 1117 bytes} [Tue Sep 13 15:54:57 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 163 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 13 15:54:57.602456 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-048c8826-f994-4016-9fef-4d3f84f47246 None None] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:57.602808 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-048c8826-f994-4016-9fef-4d3f84f47246 None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:57.603109 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-048c8826-f994-4016-9fef-4d3f84f47246 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:57.621067 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.auth.core [None req-048c8826-f994-4016-9fef-4d3f84f47246 None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81438) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:54:57.621664 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24136ff4-df49-40f8-b920-971ec5dd2f67 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:57.622272 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24136ff4-df49-40f8-b920-971ec5dd2f67 None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:57.622375 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-048c8826-f994-4016-9fef-4d3f84f47246 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:54:57.622552 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24136ff4-df49-40f8-b920-971ec5dd2f67 None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:57.622824 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24136ff4-df49-40f8-b920-971ec5dd2f67 None admin] PATH_INFO: `/v3/services` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:57.623483 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24136ff4-df49-40f8-b920-971ec5dd2f67 None admin] RBAC: Adding query filter params (name=object-store) {{(pid=81437) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:54:57.623787 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24136ff4-df49-40f8-b920-971ec5dd2f67 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:57.625147 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24136ff4-df49-40f8-b920-971ec5dd2f67 None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:57.629787 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 496/977] 10.222.0.50 () {56 vars in 1122 bytes} [Tue Sep 13 15:54:57 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 13 15:54:57.630558 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24136ff4-df49-40f8-b920-971ec5dd2f67 None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:57.683800 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:54:57.696781 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 482/978] 10.222.0.50 () {58 vars in 1024 bytes} [Tue Sep 13 15:54:57 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 13 15:54:57.697544 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-048c8826-f994-4016-9fef-4d3f84f47246 None None] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:57.747645 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24136ff4-df49-40f8-b920-971ec5dd2f67 None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:57.786013 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5d42075-14e1-4946-bc46-89c9aecd5854 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:57.786621 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5d42075-14e1-4946-bc46-89c9aecd5854 None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:57.786931 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5d42075-14e1-4946-bc46-89c9aecd5854 None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:57.787215 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5d42075-14e1-4946-bc46-89c9aecd5854 None admin] PATH_INFO: `/v3/users` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:57.787940 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5d42075-14e1-4946-bc46-89c9aecd5854 None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=81437) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:54:57.788382 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5d42075-14e1-4946-bc46-89c9aecd5854 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:57.789899 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5d42075-14e1-4946-bc46-89c9aecd5854 None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:57.805852 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-048c8826-f994-4016-9fef-4d3f84f47246 None None] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:57.807430 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 497/979] 10.222.0.50 () {56 vars in 1149 bytes} [Tue Sep 13 15:54:57 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:54:57.813084 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5d42075-14e1-4946-bc46-89c9aecd5854 None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:57.840856 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:54:57.852422 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15dd5db3-114b-430d-a6bf-558cb4230986 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:57.853306 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15dd5db3-114b-430d-a6bf-558cb4230986 None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:57.853772 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15dd5db3-114b-430d-a6bf-558cb4230986 None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:57.854228 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15dd5db3-114b-430d-a6bf-558cb4230986 None admin] PATH_INFO: `/v3/services` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:57.855072 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15dd5db3-114b-430d-a6bf-558cb4230986 None admin] RBAC: Adding query filter params (type=object-store) {{(pid=81438) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:54:57.855586 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15dd5db3-114b-430d-a6bf-558cb4230986 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:57.857199 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15dd5db3-114b-430d-a6bf-558cb4230986 None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:57.862748 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 483/980] 10.222.0.50 () {56 vars in 1122 bytes} [Tue Sep 13 15:54:57 2022] GET /identity/v3/services?type=object-store => generated 356 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:54:57.868363 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15dd5db3-114b-430d-a6bf-558cb4230986 None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:57.919457 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5d42075-14e1-4946-bc46-89c9aecd5854 None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:57.930303 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:54:57.957081 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-311ebed4-d38d-4a31-a16c-093f1402b327 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:57.957601 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-311ebed4-d38d-4a31-a16c-093f1402b327 None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:57.957882 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-311ebed4-d38d-4a31-a16c-093f1402b327 None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:57.958158 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-311ebed4-d38d-4a31-a16c-093f1402b327 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:57.982350 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15dd5db3-114b-430d-a6bf-558cb4230986 None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:58.017480 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bd08276-bdd2-4224-8a14-645b1b57c86c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:58.018083 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bd08276-bdd2-4224-8a14-645b1b57c86c None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:58.018365 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bd08276-bdd2-4224-8a14-645b1b57c86c None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:58.018636 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bd08276-bdd2-4224-8a14-645b1b57c86c None admin] PATH_INFO: `/v3/endpoints` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:58.019368 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bd08276-bdd2-4224-8a14-645b1b57c86c None admin] RBAC: Adding query filter params (interface=admin, service_id=f2c069a5bfbd4001a28d284a78e9f24d, region_id=RegionOne) {{(pid=81438) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:54:58.019679 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bd08276-bdd2-4224-8a14-645b1b57c86c None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:58.021035 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bd08276-bdd2-4224-8a14-645b1b57c86c None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:58.025474 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-311ebed4-d38d-4a31-a16c-093f1402b327 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:58.025920 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 484/981] 10.222.0.50 () {56 vars in 1249 bytes} [Tue Sep 13 15:54:57 2022] GET /identity/v3/endpoints?service_id=f2c069a5bfbd4001a28d284a78e9f24d&interface=admin®ion_id=RegionOne => generated 193 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 13 15:54:58.026684 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bd08276-bdd2-4224-8a14-645b1b57c86c None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:58.027007 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-311ebed4-d38d-4a31-a16c-093f1402b327 None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:58.052915 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:54:58.126773 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bd08276-bdd2-4224-8a14-645b1b57c86c None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:58.131742 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 498/982] 10.222.0.50 () {58 vars in 1282 bytes} [Tue Sep 13 15:54:57 2022] GET /identity/v3/auth/tokens => generated 1474 bytes in 319 msecs (HTTP/1.1 200) 6 headers in 359 bytes (2 switches on core 0) Sep 13 15:54:58.133380 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-311ebed4-d38d-4a31-a16c-093f1402b327 None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:58.147861 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:54:58.166334 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2b00558-cf3b-46de-970f-511c426f566f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:58.166981 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2b00558-cf3b-46de-970f-511c426f566f None admin] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:58.167261 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2b00558-cf3b-46de-970f-511c426f566f None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:58.167536 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2b00558-cf3b-46de-970f-511c426f566f None admin] PATH_INFO: `/v3/services` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:58.168176 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2b00558-cf3b-46de-970f-511c426f566f None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:58.169532 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2b00558-cf3b-46de-970f-511c426f566f None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:58.185240 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 485/983] 10.222.0.50 () {60 vars in 1141 bytes} [Tue Sep 13 15:54:58 2022] POST /identity/v3/services => generated 242 bytes in 159 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 15:54:58.186132 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2b00558-cf3b-46de-970f-511c426f566f None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:58.240457 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-311ebed4-d38d-4a31-a16c-093f1402b327 None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:58.269333 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:54:58.289303 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f6b50bd-9747-419e-a042-2a42871112a6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:58.289935 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f6b50bd-9747-419e-a042-2a42871112a6 None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:58.290215 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f6b50bd-9747-419e-a042-2a42871112a6 None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:58.290493 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f6b50bd-9747-419e-a042-2a42871112a6 None admin] PATH_INFO: `/v3/services` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:58.291138 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f6b50bd-9747-419e-a042-2a42871112a6 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:58.292543 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f6b50bd-9747-419e-a042-2a42871112a6 None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:58.299927 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 499/984] 10.222.0.50 () {56 vars in 1087 bytes} [Tue Sep 13 15:54:58 2022] GET /identity/v3/services => generated 1727 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 15:54:58.316036 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2b00558-cf3b-46de-970f-511c426f566f None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:58.357414 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-396916ec-9137-40ce-bd58-7533599df5d6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:58.358520 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-396916ec-9137-40ce-bd58-7533599df5d6 None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:58.359053 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-396916ec-9137-40ce-bd58-7533599df5d6 None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:58.359534 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-396916ec-9137-40ce-bd58-7533599df5d6 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:58.373025 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-396916ec-9137-40ce-bd58-7533599df5d6 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:58.374500 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-396916ec-9137-40ce-bd58-7533599df5d6 None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:58.391220 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application [None req-396916ec-9137-40ce-bd58-7533599df5d6 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:54:58.391220 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:54:58.391220 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:54:58.391220 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:54:58.391220 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:54:58.391220 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:54:58.391220 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:54:58.391220 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:54:58.391220 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:54:58.391220 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:54:58.391220 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:54:58.391220 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:54:58.391220 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 13 15:54:58.391220 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 13 15:54:58.391220 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 13 15:54:58.391220 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 13 15:54:58.392245 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:54:58.392245 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:54:58.392245 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:54:58.392245 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:54:58.392245 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:54:58.392245 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:54:58.392245 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:54:58.392245 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application with Lock( Sep 13 15:54:58.392245 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:54:58.392245 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:54:58.392245 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:54:58.392245 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:54:58.392245 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:54:58.392245 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:54:58.392245 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:54:58.392245 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:54:58.392245 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 13 15:54:58.393158 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 13 15:54:58.393158 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:54:58.393158 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Sep 13 15:54:58.393158 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 486/985] 10.222.0.50 () {56 vars in 1108 bytes} [Tue Sep 13 15:54:58 2022] GET /identity/v3/domains/Default => generated 87 bytes in 207 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:54:58.399016 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f6b50bd-9747-419e-a042-2a42871112a6 None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:58.509198 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f6b50bd-9747-419e-a042-2a42871112a6 None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:58.543886 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84a1a8ba-b600-40bc-acdb-60f325d14ac1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:58.544490 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84a1a8ba-b600-40bc-acdb-60f325d14ac1 None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:58.544765 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84a1a8ba-b600-40bc-acdb-60f325d14ac1 None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:58.545032 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84a1a8ba-b600-40bc-acdb-60f325d14ac1 None admin] PATH_INFO: `/v3/domains` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:58.545665 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84a1a8ba-b600-40bc-acdb-60f325d14ac1 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81437) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:54:58.545964 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84a1a8ba-b600-40bc-acdb-60f325d14ac1 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:58.547348 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84a1a8ba-b600-40bc-acdb-60f325d14ac1 None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:58.564288 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 500/986] 10.222.0.50 () {56 vars in 1109 bytes} [Tue Sep 13 15:54:58 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 13 15:54:58.565537 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-396916ec-9137-40ce-bd58-7533599df5d6 None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:58.597629 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:54:58.670506 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-396916ec-9137-40ce-bd58-7533599df5d6 None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:58.705551 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5e60f9a-5b35-4f3b-9b91-ba141c7a12a3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:58.706184 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5e60f9a-5b35-4f3b-9b91-ba141c7a12a3 None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:58.706473 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5e60f9a-5b35-4f3b-9b91-ba141c7a12a3 None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:58.706884 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5e60f9a-5b35-4f3b-9b91-ba141c7a12a3 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:58.714628 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5e60f9a-5b35-4f3b-9b91-ba141c7a12a3 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:58.716163 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5e60f9a-5b35-4f3b-9b91-ba141c7a12a3 None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:58.727595 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application [None req-d5e60f9a-5b35-4f3b-9b91-ba141c7a12a3 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 13 15:54:58.727595 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:54:58.727595 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:54:58.727595 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:54:58.727595 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:54:58.727595 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:54:58.727595 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:54:58.727595 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:54:58.727595 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:54:58.727595 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:54:58.727595 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:54:58.727595 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:54:58.727595 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 13 15:54:58.727595 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 13 15:54:58.727595 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 13 15:54:58.728544 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 13 15:54:58.728544 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:54:58.728544 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:54:58.728544 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:54:58.728544 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:54:58.728544 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:54:58.728544 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:54:58.728544 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:54:58.728544 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application with Lock( Sep 13 15:54:58.728544 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:54:58.728544 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:54:58.728544 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:54:58.728544 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:54:58.728544 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:54:58.728544 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:54:58.728544 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:54:58.728544 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:54:58.729485 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 13 15:54:58.729485 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 13 15:54:58.729485 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 13 15:54:58.729485 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 13 15:54:58.729485 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 13 15:54:58.729485 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 13 15:54:58.729485 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 13 15:54:58.729485 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Sep 13 15:54:58.729485 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 487/987] 10.222.0.50 () {56 vars in 1111 bytes} [Tue Sep 13 15:54:58 2022] GET /identity/v3/projects/service => generated 88 bytes in 164 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:54:58.735873 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84a1a8ba-b600-40bc-acdb-60f325d14ac1 None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:58.784896 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:54:58.844511 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84a1a8ba-b600-40bc-acdb-60f325d14ac1 None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:58.878917 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa543954-9de6-4184-8ec9-02a6848f53b9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:58.879534 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa543954-9de6-4184-8ec9-02a6848f53b9 None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:58.879814 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa543954-9de6-4184-8ec9-02a6848f53b9 None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:58.880088 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa543954-9de6-4184-8ec9-02a6848f53b9 None admin] PATH_INFO: `/v3/projects` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:58.880803 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa543954-9de6-4184-8ec9-02a6848f53b9 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81437) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:54:58.881111 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa543954-9de6-4184-8ec9-02a6848f53b9 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:58.883158 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa543954-9de6-4184-8ec9-02a6848f53b9 None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:58.899480 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 501/988] 10.222.0.50 () {56 vars in 1148 bytes} [Tue Sep 13 15:54:58 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:54:58.900932 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5e60f9a-5b35-4f3b-9b91-ba141c7a12a3 None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:54:58.997526 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:54:59.008479 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5e60f9a-5b35-4f3b-9b91-ba141c7a12a3 None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:54:59.046226 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bcb8539-8b4b-46fd-bb0a-d6f123c3788f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:54:59.046772 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bcb8539-8b4b-46fd-bb0a-d6f123c3788f None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:54:59.047086 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bcb8539-8b4b-46fd-bb0a-d6f123c3788f None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:54:59.047362 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bcb8539-8b4b-46fd-bb0a-d6f123c3788f None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:54:59.048091 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1bcb8539-8b4b-46fd-bb0a-d6f123c3788f None admin] RBAC: Adding query filter params (role.id=20e2f11234344900a51fc76fe5b625f3, scope.project.id=5811e3b83a1e462e9320cad79a3ad791, user.id=f8364b1a190d4218bf353a89bdb81448) {{(pid=81438) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:54:59.048399 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1bcb8539-8b4b-46fd-bb0a-d6f123c3788f None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:54:59.049826 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1bcb8539-8b4b-46fd-bb0a-d6f123c3788f None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:54:59.065856 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 488/989] 10.222.0.50 () {56 vars in 1374 bytes} [Tue Sep 13 15:54:58 2022] GET /identity/v3/role_assignments?user.id=f8364b1a190d4218bf353a89bdb81448&scope.project.id=5811e3b83a1e462e9320cad79a3ad791&role.id=20e2f11234344900a51fc76fe5b625f3 => generated 608 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:55:00.408343 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05a83339-c76d-4d3d-8add-cabb6061c28b None None] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:55:00.408865 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05a83339-c76d-4d3d-8add-cabb6061c28b None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:55:00.408945 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05a83339-c76d-4d3d-8add-cabb6061c28b None None] PATH_INFO: `/` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:55:00.410281 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 502/990] 10.222.0.50 () {54 vars in 927 bytes} [Tue Sep 13 15:55:00 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:55:00.416248 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d32f1949-1876-4aa3-8057-d9889dd562de None None] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:55:00.416654 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d32f1949-1876-4aa3-8057-d9889dd562de None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:55:00.416835 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d32f1949-1876-4aa3-8057-d9889dd562de None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:55:00.435300 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.auth.core [None req-d32f1949-1876-4aa3-8057-d9889dd562de None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81438) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:55:00.436489 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-d32f1949-1876-4aa3-8057-d9889dd562de None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:55:00.482881 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6af359b9-564b-43ef-a949-310d04764a54 None None] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:55:00.483220 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6af359b9-564b-43ef-a949-310d04764a54 None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:55:00.483460 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6af359b9-564b-43ef-a949-310d04764a54 None None] PATH_INFO: `/` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:55:00.485064 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 503/991] 10.222.0.50 () {54 vars in 927 bytes} [Tue Sep 13 15:55:00 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 13 15:55:00.489745 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dca73fb0-44f1-4bde-997e-b4c93a1c7f7f None None] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:55:00.490029 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dca73fb0-44f1-4bde-997e-b4c93a1c7f7f None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:55:00.490304 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dca73fb0-44f1-4bde-997e-b4c93a1c7f7f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:55:00.508007 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.auth.core [None req-dca73fb0-44f1-4bde-997e-b4c93a1c7f7f None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81437) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:55:00.509176 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-dca73fb0-44f1-4bde-997e-b4c93a1c7f7f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:55:00.509791 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 489/992] 10.222.0.50 () {58 vars in 1024 bytes} [Tue Sep 13 15:55:00 2022] POST /identity/v3/auth/tokens => generated 1570 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 13 15:55:00.577167 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 504/993] 10.222.0.50 () {58 vars in 1024 bytes} [Tue Sep 13 15:55:00 2022] POST /identity/v3/auth/tokens => generated 1570 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 13 15:55:00.779287 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d32f1949-1876-4aa3-8057-d9889dd562de None None] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:55:00.850043 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dca73fb0-44f1-4bde-997e-b4c93a1c7f7f None None] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:55:00.868038 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:55:00.887173 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:55:00.889915 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d32f1949-1876-4aa3-8057-d9889dd562de None None] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:55:00.929781 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f714086-0bfd-4273-ba44-2fbda84d80f6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:55:00.930423 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f714086-0bfd-4273-ba44-2fbda84d80f6 None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:55:00.930703 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f714086-0bfd-4273-ba44-2fbda84d80f6 None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:55:00.931011 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f714086-0bfd-4273-ba44-2fbda84d80f6 None admin] PATH_INFO: `/v3/services/compute` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:55:00.931592 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f714086-0bfd-4273-ba44-2fbda84d80f6 None admin] RBAC: Authorizing `identity:get_service(service_id=compute)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:55:00.933249 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f714086-0bfd-4273-ba44-2fbda84d80f6 None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:55:00.942385 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application [None req-5f714086-0bfd-4273-ba44-2fbda84d80f6 None admin] Could not find service: compute.: keystone.exception.ServiceNotFound: Could not find service: compute. Sep 13 15:55:00.942385 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:55:00.942385 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 13 15:55:00.942385 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 13 15:55:00.942385 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 13 15:55:00.942385 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 13 15:55:00.942385 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 13 15:55:00.942385 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 13 15:55:00.942385 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Sep 13 15:55:00.942385 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Sep 13 15:55:00.942385 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 13 15:55:00.942385 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Sep 13 15:55:00.942385 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:55:00.942385 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:55:00.942385 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:55:00.942385 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:55:00.942385 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:55:00.943433 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:55:00.943433 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:55:00.943433 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:55:00.943433 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:55:00.943433 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:55:00.943433 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:55:00.943433 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 13 15:55:00.943433 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 13 15:55:00.943433 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 13 15:55:00.943433 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 13 15:55:00.943433 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:55:00.943433 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:55:00.943433 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:55:00.943433 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:55:00.943433 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:55:00.943433 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:55:00.945479 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:55:00.945479 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application with Lock( Sep 13 15:55:00.945479 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:55:00.945479 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:55:00.945479 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:55:00.945479 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:55:00.945479 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:55:00.945479 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:55:00.945479 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:55:00.945479 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:55:00.945479 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 13 15:55:00.945479 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 13 15:55:00.945479 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Sep 13 15:55:00.945479 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Sep 13 15:55:00.945479 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 490/994] 10.222.0.50 () {56 vars in 1111 bytes} [Tue Sep 13 15:55:00 2022] GET /identity/v3/services/compute => generated 88 bytes in 165 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:55:00.949747 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f714086-0bfd-4273-ba44-2fbda84d80f6 None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:55:00.962363 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dca73fb0-44f1-4bde-997e-b4c93a1c7f7f None None] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:55:01.000063 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa8f7a27-f657-45a1-8a39-1011e9842e3b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:55:01.001170 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa8f7a27-f657-45a1-8a39-1011e9842e3b None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:55:01.001707 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa8f7a27-f657-45a1-8a39-1011e9842e3b None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:55:01.002205 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa8f7a27-f657-45a1-8a39-1011e9842e3b None admin] PATH_INFO: `/v3/services/object-store` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:55:01.003045 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa8f7a27-f657-45a1-8a39-1011e9842e3b None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:55:01.004755 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa8f7a27-f657-45a1-8a39-1011e9842e3b None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:55:01.015659 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application [None req-fa8f7a27-f657-45a1-8a39-1011e9842e3b None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Sep 13 15:55:01.015659 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:55:01.015659 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 13 15:55:01.015659 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 13 15:55:01.015659 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 13 15:55:01.015659 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 13 15:55:01.015659 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 13 15:55:01.015659 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 13 15:55:01.015659 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Sep 13 15:55:01.015659 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application Sep 13 15:55:01.015659 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 13 15:55:01.015659 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application Sep 13 15:55:01.015659 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:55:01.015659 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:55:01.015659 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:55:01.015659 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:55:01.015659 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:55:01.016678 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:55:01.016678 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:55:01.016678 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:55:01.016678 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:55:01.016678 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:55:01.016678 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:55:01.016678 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 13 15:55:01.016678 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 13 15:55:01.016678 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 13 15:55:01.016678 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 13 15:55:01.016678 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:55:01.016678 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:55:01.016678 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:55:01.016678 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:55:01.016678 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:55:01.016678 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:55:01.017547 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:55:01.017547 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application with Lock( Sep 13 15:55:01.017547 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:55:01.017547 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:55:01.017547 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:55:01.017547 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:55:01.017547 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:55:01.017547 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:55:01.017547 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:55:01.017547 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:55:01.017547 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 13 15:55:01.017547 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 13 15:55:01.017547 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Sep 13 15:55:01.017547 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application Sep 13 15:55:01.020478 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 505/995] 10.222.0.50 () {56 vars in 1126 bytes} [Tue Sep 13 15:55:00 2022] GET /identity/v3/services/object-store => generated 93 bytes in 171 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:55:01.025775 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa8f7a27-f657-45a1-8a39-1011e9842e3b None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:55:01.060056 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f714086-0bfd-4273-ba44-2fbda84d80f6 None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:55:01.099624 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5870e987-71ac-4ff0-8779-6938f59743e2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:55:01.100244 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5870e987-71ac-4ff0-8779-6938f59743e2 None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:55:01.100793 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5870e987-71ac-4ff0-8779-6938f59743e2 None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:55:01.101073 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5870e987-71ac-4ff0-8779-6938f59743e2 None admin] PATH_INFO: `/v3/services` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:55:01.102196 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5870e987-71ac-4ff0-8779-6938f59743e2 None admin] RBAC: Adding query filter params (name=compute) {{(pid=81438) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:55:01.102512 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5870e987-71ac-4ff0-8779-6938f59743e2 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:55:01.104194 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5870e987-71ac-4ff0-8779-6938f59743e2 None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:55:01.110263 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 491/996] 10.222.0.50 () {56 vars in 1112 bytes} [Tue Sep 13 15:55:00 2022] GET /identity/v3/services?name=compute => generated 124 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 13 15:55:01.115734 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5870e987-71ac-4ff0-8779-6938f59743e2 None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:55:01.134573 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:55:01.136352 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:55:01.139176 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa8f7a27-f657-45a1-8a39-1011e9842e3b None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:55:01.178112 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2482a00c-95e9-44f9-a951-1b44e2b6ff0d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:55:01.179295 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2482a00c-95e9-44f9-a951-1b44e2b6ff0d None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:55:01.179584 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2482a00c-95e9-44f9-a951-1b44e2b6ff0d None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:55:01.179856 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2482a00c-95e9-44f9-a951-1b44e2b6ff0d None admin] PATH_INFO: `/v3/services` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:55:01.180480 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2482a00c-95e9-44f9-a951-1b44e2b6ff0d None admin] RBAC: Adding query filter params (name=object-store) {{(pid=81437) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:55:01.180790 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2482a00c-95e9-44f9-a951-1b44e2b6ff0d None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:55:01.182466 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2482a00c-95e9-44f9-a951-1b44e2b6ff0d None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:55:01.187413 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 506/997] 10.222.0.50 () {56 vars in 1122 bytes} [Tue Sep 13 15:55:01 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:55:01.192983 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2482a00c-95e9-44f9-a951-1b44e2b6ff0d None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:55:01.243582 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5870e987-71ac-4ff0-8779-6938f59743e2 None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:55:01.278590 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-288d79bd-3cf1-4106-932e-23facc9775fc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:55:01.279245 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-288d79bd-3cf1-4106-932e-23facc9775fc None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:55:01.279524 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-288d79bd-3cf1-4106-932e-23facc9775fc None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:55:01.279803 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-288d79bd-3cf1-4106-932e-23facc9775fc None admin] PATH_INFO: `/v3/services` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:55:01.280423 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-288d79bd-3cf1-4106-932e-23facc9775fc None admin] RBAC: Adding query filter params (type=compute) {{(pid=81438) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:55:01.280725 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-288d79bd-3cf1-4106-932e-23facc9775fc None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:55:01.282108 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-288d79bd-3cf1-4106-932e-23facc9775fc None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:55:01.286968 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 492/998] 10.222.0.50 () {56 vars in 1112 bytes} [Tue Sep 13 15:55:01 2022] GET /identity/v3/services?type=compute => generated 352 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:55:01.288729 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d410ab43-bfaa-4867-8939-c48f75fd7f76 None None] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:55:01.289009 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d410ab43-bfaa-4867-8939-c48f75fd7f76 None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:55:01.289286 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d410ab43-bfaa-4867-8939-c48f75fd7f76 None None] PATH_INFO: `/` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:55:01.290474 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 493/999] 10.222.0.50 () {54 vars in 927 bytes} [Tue Sep 13 15:55:01 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:55:01.293204 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2482a00c-95e9-44f9-a951-1b44e2b6ff0d None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:55:01.295296 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d410ab43-bfaa-4867-8939-c48f75fd7f76 None None] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:55:01.327589 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-172204bb-1022-4af7-bb18-131ca6c259b7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:55:01.328194 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-172204bb-1022-4af7-bb18-131ca6c259b7 None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:55:01.328477 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-172204bb-1022-4af7-bb18-131ca6c259b7 None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:55:01.328751 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-172204bb-1022-4af7-bb18-131ca6c259b7 None admin] PATH_INFO: `/v3/services` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:55:01.329377 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-172204bb-1022-4af7-bb18-131ca6c259b7 None admin] RBAC: Adding query filter params (type=object-store) {{(pid=81437) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:55:01.329682 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-172204bb-1022-4af7-bb18-131ca6c259b7 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:55:01.331109 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-172204bb-1022-4af7-bb18-131ca6c259b7 None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:55:01.336716 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 507/1000] 10.222.0.50 () {56 vars in 1122 bytes} [Tue Sep 13 15:55:01 2022] GET /identity/v3/services?type=object-store => generated 356 bytes in 144 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 13 15:55:01.338015 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a05214f4-b414-49e1-8770-7c2fd43a2cdf None None] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:55:01.338307 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a05214f4-b414-49e1-8770-7c2fd43a2cdf None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:55:01.338589 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a05214f4-b414-49e1-8770-7c2fd43a2cdf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:55:01.356638 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.auth.core [None req-a05214f4-b414-49e1-8770-7c2fd43a2cdf None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81437) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:55:01.357820 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-a05214f4-b414-49e1-8770-7c2fd43a2cdf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:55:01.373066 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:55:01.378110 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:55:01.398283 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d410ab43-bfaa-4867-8939-c48f75fd7f76 None None] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:55:01.425029 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 508/1001] 10.222.0.50 () {58 vars in 1024 bytes} [Tue Sep 13 15:55:01 2022] POST /identity/v3/auth/tokens => generated 1570 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 13 15:55:01.425675 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a05214f4-b414-49e1-8770-7c2fd43a2cdf None None] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:55:01.434674 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bce29e5c-92fb-4961-942e-2153873503f2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:55:01.435334 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bce29e5c-92fb-4961-942e-2153873503f2 None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:55:01.435620 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bce29e5c-92fb-4961-942e-2153873503f2 None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:55:01.435881 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bce29e5c-92fb-4961-942e-2153873503f2 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:55:01.436918 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bce29e5c-92fb-4961-942e-2153873503f2 None admin] RBAC: Adding query filter params (interface=public, service_id=609bbd53df9b4f7485a2b1be9c28db19, region_id=RegionOne) {{(pid=81438) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:55:01.437219 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bce29e5c-92fb-4961-942e-2153873503f2 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:55:01.438877 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bce29e5c-92fb-4961-942e-2153873503f2 None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:55:01.444885 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 494/1002] 10.222.0.50 () {56 vars in 1251 bytes} [Tue Sep 13 15:55:01 2022] GET /identity/v3/endpoints?service_id=609bbd53df9b4f7485a2b1be9c28db19&interface=public®ion_id=RegionOne => generated 194 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:55:01.450820 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bce29e5c-92fb-4961-942e-2153873503f2 None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:55:01.532896 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a05214f4-b414-49e1-8770-7c2fd43a2cdf None None] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:55:01.560053 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bce29e5c-92fb-4961-942e-2153873503f2 None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:55:01.567782 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99e7b40a-2acb-458d-a6b2-bfbbc1650f59 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:55:01.568385 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99e7b40a-2acb-458d-a6b2-bfbbc1650f59 None admin] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:55:01.568662 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99e7b40a-2acb-458d-a6b2-bfbbc1650f59 None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:55:01.568935 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99e7b40a-2acb-458d-a6b2-bfbbc1650f59 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:55:01.570204 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99e7b40a-2acb-458d-a6b2-bfbbc1650f59 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:55:01.571602 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99e7b40a-2acb-458d-a6b2-bfbbc1650f59 None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:55:01.589189 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 509/1003] 10.222.0.50 () {60 vars in 1144 bytes} [Tue Sep 13 15:55:01 2022] POST /identity/v3/endpoints => generated 325 bytes in 164 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 15:55:01.617072 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-accb569d-a11c-418c-9acb-6c278528f08b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:55:01.617707 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-accb569d-a11c-418c-9acb-6c278528f08b None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:55:01.617998 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-accb569d-a11c-418c-9acb-6c278528f08b None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:55:01.618256 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-accb569d-a11c-418c-9acb-6c278528f08b None admin] PATH_INFO: `/v3/services` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:55:01.618965 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-accb569d-a11c-418c-9acb-6c278528f08b None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:55:01.620372 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-accb569d-a11c-418c-9acb-6c278528f08b None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:55:01.623069 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:55:01.628822 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 495/1004] 10.222.0.50 () {56 vars in 1087 bytes} [Tue Sep 13 15:55:01 2022] GET /identity/v3/services => generated 1727 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 13 15:55:01.691318 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99e7b40a-2acb-458d-a6b2-bfbbc1650f59 None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:55:01.713752 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:55:01.812038 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99e7b40a-2acb-458d-a6b2-bfbbc1650f59 None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:55:01.850316 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1a113ca-ae0e-4a3a-ab20-7aa8b19487b9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:55:01.851609 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1a113ca-ae0e-4a3a-ab20-7aa8b19487b9 None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:55:01.851977 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1a113ca-ae0e-4a3a-ab20-7aa8b19487b9 None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:55:01.852257 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1a113ca-ae0e-4a3a-ab20-7aa8b19487b9 None admin] PATH_INFO: `/v3/services/image` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:55:01.852845 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1a113ca-ae0e-4a3a-ab20-7aa8b19487b9 None admin] RBAC: Authorizing `identity:get_service(service_id=image)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:55:01.854230 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1a113ca-ae0e-4a3a-ab20-7aa8b19487b9 None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:55:01.863279 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application [None req-a1a113ca-ae0e-4a3a-ab20-7aa8b19487b9 None admin] Could not find service: image.: keystone.exception.ServiceNotFound: Could not find service: image. Sep 13 15:55:01.863279 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:55:01.863279 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 13 15:55:01.863279 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 13 15:55:01.863279 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 13 15:55:01.863279 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 13 15:55:01.863279 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 13 15:55:01.863279 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 13 15:55:01.863279 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Sep 13 15:55:01.863279 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application Sep 13 15:55:01.863279 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 13 15:55:01.863279 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application Sep 13 15:55:01.863279 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:55:01.863279 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:55:01.863279 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:55:01.863279 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:55:01.863279 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:55:01.864514 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:55:01.864514 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:55:01.864514 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:55:01.864514 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:55:01.864514 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:55:01.864514 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:55:01.864514 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 13 15:55:01.864514 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 13 15:55:01.864514 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 13 15:55:01.864514 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 13 15:55:01.864514 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:55:01.864514 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:55:01.864514 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:55:01.864514 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:55:01.864514 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:55:01.864514 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:55:01.865668 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:55:01.865668 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application with Lock( Sep 13 15:55:01.865668 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:55:01.865668 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:55:01.865668 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:55:01.865668 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:55:01.865668 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:55:01.865668 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:55:01.865668 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:55:01.865668 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:55:01.865668 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 13 15:55:01.865668 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 13 15:55:01.865668 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Sep 13 15:55:01.865668 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application Sep 13 15:55:01.865668 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 510/1005] 10.222.0.50 () {56 vars in 1105 bytes} [Tue Sep 13 15:55:01 2022] GET /identity/v3/services/image => generated 86 bytes in 174 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:55:01.871655 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-accb569d-a11c-418c-9acb-6c278528f08b None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:55:01.975184 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-accb569d-a11c-418c-9acb-6c278528f08b None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:55:02.009407 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50e872ae-9133-4f78-b719-e0bab2b517a9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:55:02.010022 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50e872ae-9133-4f78-b719-e0bab2b517a9 None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:55:02.010300 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50e872ae-9133-4f78-b719-e0bab2b517a9 None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:55:02.010570 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50e872ae-9133-4f78-b719-e0bab2b517a9 None admin] PATH_INFO: `/v3/services` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:55:02.011230 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50e872ae-9133-4f78-b719-e0bab2b517a9 None admin] RBAC: Adding query filter params (name=image) {{(pid=81438) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:55:02.011534 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50e872ae-9133-4f78-b719-e0bab2b517a9 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:55:02.012911 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50e872ae-9133-4f78-b719-e0bab2b517a9 None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:55:02.022259 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 496/1006] 10.222.0.50 () {56 vars in 1108 bytes} [Tue Sep 13 15:55:01 2022] GET /identity/v3/services?name=image => generated 122 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:55:02.024407 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1a113ca-ae0e-4a3a-ab20-7aa8b19487b9 None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:55:02.113783 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:55:02.135794 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1a113ca-ae0e-4a3a-ab20-7aa8b19487b9 None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:55:02.171162 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1962798-c451-4e41-a393-051774534a5f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:55:02.171772 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1962798-c451-4e41-a393-051774534a5f None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:55:02.172048 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1962798-c451-4e41-a393-051774534a5f None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:55:02.172317 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1962798-c451-4e41-a393-051774534a5f None admin] PATH_INFO: `/v3/services` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:55:02.172941 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1962798-c451-4e41-a393-051774534a5f None admin] RBAC: Adding query filter params (type=image) {{(pid=81437) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:55:02.173244 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1962798-c451-4e41-a393-051774534a5f None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:55:02.174606 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1962798-c451-4e41-a393-051774534a5f None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:55:02.179763 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 511/1007] 10.222.0.50 () {56 vars in 1108 bytes} [Tue Sep 13 15:55:02 2022] GET /identity/v3/services?type=image => generated 122 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 13 15:55:03.811456 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a90059bd-f623-4490-ae6c-c8bca6742ef5 None None] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:55:03.811986 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a90059bd-f623-4490-ae6c-c8bca6742ef5 None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:55:03.811986 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a90059bd-f623-4490-ae6c-c8bca6742ef5 None None] PATH_INFO: `/` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:55:03.813464 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 497/1008] 10.222.0.50 () {54 vars in 927 bytes} [Tue Sep 13 15:55:03 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:55:03.818560 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73cd9651-6322-4627-a483-376002ac5730 None None] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:55:03.819028 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73cd9651-6322-4627-a483-376002ac5730 None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:55:03.819184 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73cd9651-6322-4627-a483-376002ac5730 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:55:03.832178 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e68bc1e-5c01-4c5b-a97d-a4927be920ee None None] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:55:03.832456 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e68bc1e-5c01-4c5b-a97d-a4927be920ee None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:55:03.832722 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e68bc1e-5c01-4c5b-a97d-a4927be920ee None None] PATH_INFO: `/` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:55:03.834146 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 498/1009] 10.222.0.50 () {54 vars in 927 bytes} [Tue Sep 13 15:55:03 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 13 15:55:03.837775 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.auth.core [None req-73cd9651-6322-4627-a483-376002ac5730 None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81437) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:55:03.838815 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-73cd9651-6322-4627-a483-376002ac5730 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:55:03.838956 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0231d9b0-df0b-43b6-993e-acbf0ab7406a None None] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:55:03.839240 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0231d9b0-df0b-43b6-993e-acbf0ab7406a None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:55:03.839512 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0231d9b0-df0b-43b6-993e-acbf0ab7406a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:55:03.860420 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.auth.core [None req-0231d9b0-df0b-43b6-993e-acbf0ab7406a None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81438) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:55:03.861598 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-0231d9b0-df0b-43b6-993e-acbf0ab7406a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:55:03.919968 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 512/1010] 10.222.0.50 () {58 vars in 1024 bytes} [Tue Sep 13 15:55:03 2022] POST /identity/v3/auth/tokens => generated 1717 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 13 15:55:03.927803 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:55:03.938374 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 499/1011] 10.222.0.50 () {58 vars in 1024 bytes} [Tue Sep 13 15:55:03 2022] POST /identity/v3/auth/tokens => generated 1717 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 13 15:55:04.201603 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73cd9651-6322-4627-a483-376002ac5730 None None] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:55:04.209001 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0231d9b0-df0b-43b6-993e-acbf0ab7406a None None] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:55:04.294021 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:55:04.320032 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0231d9b0-df0b-43b6-993e-acbf0ab7406a None None] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:55:04.323128 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73cd9651-6322-4627-a483-376002ac5730 None None] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:55:04.355202 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78e4ccef-d91d-4111-8545-f1b24a28ed84 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:55:04.355840 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78e4ccef-d91d-4111-8545-f1b24a28ed84 None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:55:04.356119 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78e4ccef-d91d-4111-8545-f1b24a28ed84 None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:55:04.356393 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78e4ccef-d91d-4111-8545-f1b24a28ed84 None admin] PATH_INFO: `/v3/services/compute` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:55:04.356969 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78e4ccef-d91d-4111-8545-f1b24a28ed84 None admin] RBAC: Authorizing `identity:get_service(service_id=compute)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:55:04.358325 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78e4ccef-d91d-4111-8545-f1b24a28ed84 None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:55:04.359168 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5b54aa0-4359-4f68-bb33-28417e947bc2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:55:04.359797 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5b54aa0-4359-4f68-bb33-28417e947bc2 None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:55:04.360077 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5b54aa0-4359-4f68-bb33-28417e947bc2 None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:55:04.360345 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5b54aa0-4359-4f68-bb33-28417e947bc2 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:55:04.361436 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5b54aa0-4359-4f68-bb33-28417e947bc2 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:55:04.362855 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5b54aa0-4359-4f68-bb33-28417e947bc2 None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:55:04.366942 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 513/1012] 10.222.0.50 () {56 vars in 1108 bytes} [Tue Sep 13 15:55:04 2022] GET /identity/v3/domains/default => generated 199 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:55:04.366942 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8bbb9a0-2886-4e48-a94c-965e4cdfc107 None None] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:55:04.366942 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8bbb9a0-2886-4e48-a94c-965e4cdfc107 None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:55:04.366942 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8bbb9a0-2886-4e48-a94c-965e4cdfc107 None None] PATH_INFO: `/` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:55:04.367764 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application [None req-78e4ccef-d91d-4111-8545-f1b24a28ed84 None admin] Could not find service: compute.: keystone.exception.ServiceNotFound: Could not find service: compute. Sep 13 15:55:04.367764 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:55:04.367764 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 13 15:55:04.367764 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 13 15:55:04.367764 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 13 15:55:04.367764 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 13 15:55:04.367764 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 13 15:55:04.367764 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 13 15:55:04.367764 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Sep 13 15:55:04.367764 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Sep 13 15:55:04.367764 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 13 15:55:04.367764 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Sep 13 15:55:04.367764 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:55:04.367764 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:55:04.367764 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:55:04.367764 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:55:04.367764 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:55:04.368723 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:55:04.368723 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:55:04.368723 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:55:04.368723 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:55:04.368723 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:55:04.368723 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:55:04.368723 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 13 15:55:04.368723 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 13 15:55:04.368723 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 13 15:55:04.368723 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 13 15:55:04.368723 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:55:04.368723 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:55:04.368723 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:55:04.368723 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:55:04.368723 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:55:04.368723 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:55:04.374706 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:55:04.374706 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application with Lock( Sep 13 15:55:04.374706 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:55:04.374706 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:55:04.374706 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:55:04.374706 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:55:04.374706 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:55:04.374706 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:55:04.374706 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:55:04.374706 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:55:04.374706 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 13 15:55:04.374706 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 13 15:55:04.374706 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Sep 13 15:55:04.374706 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Sep 13 15:55:04.374706 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 500/1013] 10.222.0.50 () {56 vars in 1111 bytes} [Tue Sep 13 15:55:04 2022] GET /identity/v3/services/compute => generated 88 bytes in 160 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 13 15:55:04.375745 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78e4ccef-d91d-4111-8545-f1b24a28ed84 None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:55:04.375830 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 514/1014] 10.222.0.50 () {54 vars in 927 bytes} [Tue Sep 13 15:55:04 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:55:04.375830 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40681a5a-846a-4fcf-99f4-bee517b92866 None None] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:55:04.375830 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40681a5a-846a-4fcf-99f4-bee517b92866 None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:55:04.375830 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40681a5a-846a-4fcf-99f4-bee517b92866 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:55:04.391475 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.auth.core [None req-40681a5a-846a-4fcf-99f4-bee517b92866 None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81437) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:55:04.392652 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-40681a5a-846a-4fcf-99f4-bee517b92866 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:55:04.432793 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:55:04.458985 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 515/1015] 10.222.0.50 () {58 vars in 1024 bytes} [Tue Sep 13 15:55:04 2022] POST /identity/v3/auth/tokens => generated 1717 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 13 15:55:04.459961 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40681a5a-846a-4fcf-99f4-bee517b92866 None None] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:55:04.474636 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78e4ccef-d91d-4111-8545-f1b24a28ed84 None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:55:04.510878 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09af2c24-306b-44c9-806e-4e34174ff783 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:55:04.511782 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09af2c24-306b-44c9-806e-4e34174ff783 None admin] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:55:04.512592 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09af2c24-306b-44c9-806e-4e34174ff783 None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:55:04.512900 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09af2c24-306b-44c9-806e-4e34174ff783 None admin] PATH_INFO: `/v3/projects` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:55:04.513552 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09af2c24-306b-44c9-806e-4e34174ff783 None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:55:04.515703 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09af2c24-306b-44c9-806e-4e34174ff783 None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:55:04.544518 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 501/1016] 10.222.0.50 () {60 vars in 1141 bytes} [Tue Sep 13 15:55:04 2022] POST /identity/v3/projects => generated 311 bytes in 174 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 15:55:04.545901 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:55:04.584065 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40681a5a-846a-4fcf-99f4-bee517b92866 None None] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:55:04.625559 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98ff6e4e-9b8b-4a47-8e02-39049748207b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:55:04.626193 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98ff6e4e-9b8b-4a47-8e02-39049748207b None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:55:04.626490 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98ff6e4e-9b8b-4a47-8e02-39049748207b None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:55:04.626760 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98ff6e4e-9b8b-4a47-8e02-39049748207b None admin] PATH_INFO: `/v3/services` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:55:04.627430 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98ff6e4e-9b8b-4a47-8e02-39049748207b None admin] RBAC: Adding query filter params (name=compute) {{(pid=81437) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:55:04.627784 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98ff6e4e-9b8b-4a47-8e02-39049748207b None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:55:04.629234 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98ff6e4e-9b8b-4a47-8e02-39049748207b None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:55:04.639172 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 516/1017] 10.222.0.50 () {56 vars in 1112 bytes} [Tue Sep 13 15:55:04 2022] GET /identity/v3/services?name=compute => generated 124 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:55:04.641462 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09af2c24-306b-44c9-806e-4e34174ff783 None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:55:04.731274 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98ff6e4e-9b8b-4a47-8e02-39049748207b None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:55:04.760736 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09af2c24-306b-44c9-806e-4e34174ff783 None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:55:04.774382 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:55:04.802949 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee510122-8b1b-4baa-ab2b-81a0a86027fe None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:55:04.803649 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee510122-8b1b-4baa-ab2b-81a0a86027fe None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:55:04.803928 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee510122-8b1b-4baa-ab2b-81a0a86027fe None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:55:04.804199 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee510122-8b1b-4baa-ab2b-81a0a86027fe None admin] PATH_INFO: `/v3/services` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:55:04.804842 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee510122-8b1b-4baa-ab2b-81a0a86027fe None admin] RBAC: Adding query filter params (type=compute) {{(pid=81438) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:55:04.805145 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee510122-8b1b-4baa-ab2b-81a0a86027fe None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:55:04.807289 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee510122-8b1b-4baa-ab2b-81a0a86027fe None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:55:04.814937 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 502/1018] 10.222.0.50 () {56 vars in 1112 bytes} [Tue Sep 13 15:55:04 2022] GET /identity/v3/services?type=compute => generated 352 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:55:04.820419 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee510122-8b1b-4baa-ab2b-81a0a86027fe None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:55:04.854027 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98ff6e4e-9b8b-4a47-8e02-39049748207b None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:55:04.890626 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e81f978-28d1-487e-9582-9f56aa8e578b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:55:04.891509 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e81f978-28d1-487e-9582-9f56aa8e578b None admin] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:55:04.891868 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e81f978-28d1-487e-9582-9f56aa8e578b None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:55:04.892148 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e81f978-28d1-487e-9582-9f56aa8e578b None admin] PATH_INFO: `/v3/services` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:55:04.892821 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e81f978-28d1-487e-9582-9f56aa8e578b None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:55:04.894299 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e81f978-28d1-487e-9582-9f56aa8e578b None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:55:04.899874 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:55:04.913998 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 517/1019] 10.222.0.50 () {60 vars in 1141 bytes} [Tue Sep 13 15:55:04 2022] POST /identity/v3/services => generated 242 bytes in 183 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 13 15:55:04.934763 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee510122-8b1b-4baa-ab2b-81a0a86027fe None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:55:04.981435 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b398db7b-82b6-4cec-87eb-0ada25f0d364 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:55:04.982059 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b398db7b-82b6-4cec-87eb-0ada25f0d364 None admin] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:55:04.982338 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b398db7b-82b6-4cec-87eb-0ada25f0d364 None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:55:04.982609 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b398db7b-82b6-4cec-87eb-0ada25f0d364 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:55:04.983312 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b398db7b-82b6-4cec-87eb-0ada25f0d364 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:55:04.984694 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b398db7b-82b6-4cec-87eb-0ada25f0d364 None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:55:05.011120 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 503/1020] 10.222.0.50 () {60 vars in 1144 bytes} [Tue Sep 13 15:55:04 2022] POST /identity/v3/endpoints => generated 334 bytes in 186 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 13 15:55:06.878917 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83425445-485b-4786-b35a-9f800ba73c78 None None] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:55:06.879444 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83425445-485b-4786-b35a-9f800ba73c78 None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:55:06.879559 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83425445-485b-4786-b35a-9f800ba73c78 None None] PATH_INFO: `/` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:55:06.880854 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 518/1021] 10.222.0.50 () {54 vars in 927 bytes} [Tue Sep 13 15:55:06 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:55:06.886421 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4232edd-7b0c-4a6e-8b77-155e04f067a4 None None] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:55:06.886814 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4232edd-7b0c-4a6e-8b77-155e04f067a4 None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:55:06.887022 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4232edd-7b0c-4a6e-8b77-155e04f067a4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:55:06.906111 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.auth.core [None req-c4232edd-7b0c-4a6e-8b77-155e04f067a4 None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81438) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:55:06.907331 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-c4232edd-7b0c-4a6e-8b77-155e04f067a4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:55:06.929373 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:55:06.985703 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 504/1022] 10.222.0.50 () {58 vars in 1024 bytes} [Tue Sep 13 15:55:06 2022] POST /identity/v3/auth/tokens => generated 1969 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 13 15:55:07.199604 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81daca31-0426-4a3f-ae33-87339e91fac4 None None] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:55:07.199808 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81daca31-0426-4a3f-ae33-87339e91fac4 None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:55:07.200076 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81daca31-0426-4a3f-ae33-87339e91fac4 None None] PATH_INFO: `/` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:55:07.201610 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 519/1023] 10.222.0.50 () {54 vars in 927 bytes} [Tue Sep 13 15:55:07 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:55:07.207284 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e98fd4c-627f-4114-af99-952cbceddabc None None] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:55:07.207583 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e98fd4c-627f-4114-af99-952cbceddabc None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:55:07.207836 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e98fd4c-627f-4114-af99-952cbceddabc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:55:07.228557 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.auth.core [None req-3e98fd4c-627f-4114-af99-952cbceddabc None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81438) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:55:07.230152 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-3e98fd4c-627f-4114-af99-952cbceddabc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:55:07.257456 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81daca31-0426-4a3f-ae33-87339e91fac4 None None] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:55:07.302946 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 505/1024] 10.222.0.50 () {58 vars in 1024 bytes} [Tue Sep 13 15:55:07 2022] POST /identity/v3/auth/tokens => generated 1969 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 13 15:55:07.315789 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1a8392d-c7fb-405d-87d8-4833b817b2de None None] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:55:07.316080 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1a8392d-c7fb-405d-87d8-4833b817b2de None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:55:07.316362 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1a8392d-c7fb-405d-87d8-4833b817b2de None None] PATH_INFO: `/` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:55:07.317790 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 506/1025] 10.222.0.50 () {54 vars in 927 bytes} [Tue Sep 13 15:55:07 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 13 15:55:07.323361 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0e0d4b1-95c0-407a-96bc-9c6d2d6d718b None None] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:55:07.323815 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0e0d4b1-95c0-407a-96bc-9c6d2d6d718b None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:55:07.324002 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0e0d4b1-95c0-407a-96bc-9c6d2d6d718b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:55:07.343227 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.auth.core [None req-c0e0d4b1-95c0-407a-96bc-9c6d2d6d718b None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81438) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:55:07.344410 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-c0e0d4b1-95c0-407a-96bc-9c6d2d6d718b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:55:07.364296 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81daca31-0426-4a3f-ae33-87339e91fac4 None None] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:55:07.401804 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7a3f038-267f-4fbe-9873-4fd4181757b4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:55:07.402434 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7a3f038-267f-4fbe-9873-4fd4181757b4 None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:55:07.402716 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7a3f038-267f-4fbe-9873-4fd4181757b4 None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:55:07.403014 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7a3f038-267f-4fbe-9873-4fd4181757b4 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:55:07.404123 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7a3f038-267f-4fbe-9873-4fd4181757b4 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:55:07.405560 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7a3f038-267f-4fbe-9873-4fd4181757b4 None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:55:07.407340 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 520/1026] 10.222.0.50 () {56 vars in 1108 bytes} [Tue Sep 13 15:55:07 2022] GET /identity/v3/domains/default => generated 199 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 13 15:55:07.412871 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7a3f038-267f-4fbe-9873-4fd4181757b4 None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:55:07.414933 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 507/1027] 10.222.0.50 () {58 vars in 1024 bytes} [Tue Sep 13 15:55:07 2022] POST /identity/v3/auth/tokens => generated 1969 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 13 15:55:07.480311 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:55:07.519489 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7a3f038-267f-4fbe-9873-4fd4181757b4 None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:55:07.553404 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3007b11-ac99-42cb-ac80-db10f091d75b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:55:07.554005 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3007b11-ac99-42cb-ac80-db10f091d75b None admin] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:55:07.554280 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3007b11-ac99-42cb-ac80-db10f091d75b None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:55:07.554553 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3007b11-ac99-42cb-ac80-db10f091d75b None admin] PATH_INFO: `/v3/users` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:55:07.555228 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3007b11-ac99-42cb-ac80-db10f091d75b None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:55:07.556773 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3007b11-ac99-42cb-ac80-db10f091d75b None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:55:07.574751 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0e0d4b1-95c0-407a-96bc-9c6d2d6d718b None None] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:55:07.587296 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 521/1028] 10.222.0.50 () {60 vars in 1132 bytes} [Tue Sep 13 15:55:07 2022] POST /identity/v3/users => generated 285 bytes in 175 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 13 15:55:07.593359 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:55:07.687765 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3007b11-ac99-42cb-ac80-db10f091d75b None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:55:07.691744 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0e0d4b1-95c0-407a-96bc-9c6d2d6d718b None None] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:55:07.733357 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08431c17-8d26-4a14-b46e-9c6679ddf0a5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:55:07.733925 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08431c17-8d26-4a14-b46e-9c6679ddf0a5 None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:55:07.734204 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08431c17-8d26-4a14-b46e-9c6679ddf0a5 None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:55:07.734477 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08431c17-8d26-4a14-b46e-9c6679ddf0a5 None admin] PATH_INFO: `/v3/services/image` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:55:07.735104 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08431c17-8d26-4a14-b46e-9c6679ddf0a5 None admin] RBAC: Authorizing `identity:get_service(service_id=image)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:55:07.736499 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08431c17-8d26-4a14-b46e-9c6679ddf0a5 None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:55:07.746514 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application [None req-08431c17-8d26-4a14-b46e-9c6679ddf0a5 None admin] Could not find service: image.: keystone.exception.ServiceNotFound: Could not find service: image. Sep 13 15:55:07.746514 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:55:07.746514 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 13 15:55:07.746514 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 13 15:55:07.746514 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 13 15:55:07.746514 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 13 15:55:07.746514 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 13 15:55:07.746514 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 13 15:55:07.746514 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Sep 13 15:55:07.746514 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Sep 13 15:55:07.746514 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 13 15:55:07.746514 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Sep 13 15:55:07.746514 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:55:07.746514 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:55:07.746514 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:55:07.746514 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:55:07.746514 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:55:07.747712 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:55:07.747712 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:55:07.747712 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:55:07.747712 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:55:07.747712 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:55:07.747712 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:55:07.747712 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 13 15:55:07.747712 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 13 15:55:07.747712 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 13 15:55:07.747712 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 13 15:55:07.747712 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:55:07.747712 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:55:07.747712 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:55:07.747712 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:55:07.747712 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:55:07.747712 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:55:07.748570 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:55:07.748570 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application with Lock( Sep 13 15:55:07.748570 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:55:07.748570 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:55:07.748570 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:55:07.748570 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:55:07.748570 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:55:07.748570 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:55:07.748570 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:55:07.748570 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:55:07.748570 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 13 15:55:07.748570 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 13 15:55:07.748570 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Sep 13 15:55:07.748570 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Sep 13 15:55:07.748570 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 508/1029] 10.222.0.50 () {56 vars in 1105 bytes} [Tue Sep 13 15:55:07 2022] GET /identity/v3/services/image => generated 86 bytes in 173 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 13 15:55:07.754839 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08431c17-8d26-4a14-b46e-9c6679ddf0a5 None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:55:07.811632 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3007b11-ac99-42cb-ac80-db10f091d75b None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:55:07.836434 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:55:07.851405 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:55:07.856804 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4448081b-0c83-442b-ad14-fdc2115a9ce8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:55:07.857521 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4448081b-0c83-442b-ad14-fdc2115a9ce8 None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:55:07.857708 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4448081b-0c83-442b-ad14-fdc2115a9ce8 None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:55:07.857981 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4448081b-0c83-442b-ad14-fdc2115a9ce8 None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:55:07.870161 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4448081b-0c83-442b-ad14-fdc2115a9ce8 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 13 15:55:07.874980 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application [None req-4448081b-0c83-442b-ad14-fdc2115a9ce8 None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Sep 13 15:55:07.874980 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:55:07.874980 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:55:07.874980 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:55:07.874980 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:55:07.874980 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:55:07.874980 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:55:07.874980 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:55:07.874980 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:55:07.874980 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:55:07.874980 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:55:07.874980 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:55:07.874980 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 13 15:55:07.874980 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 13 15:55:07.874980 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 13 15:55:07.874980 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 13 15:55:07.875978 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 13 15:55:07.875978 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 13 15:55:07.875978 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 13 15:55:07.875978 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 13 15:55:07.875978 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:55:07.875978 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:55:07.875978 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:55:07.875978 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:55:07.875978 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:55:07.875978 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:55:07.875978 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:55:07.875978 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application with Lock( Sep 13 15:55:07.875978 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:55:07.875978 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:55:07.875978 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:55:07.875978 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:55:07.877373 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:55:07.877373 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:55:07.877373 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:55:07.877373 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:55:07.877373 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 13 15:55:07.877373 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 13 15:55:07.877373 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 13 15:55:07.877373 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 13 15:55:07.877373 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 13 15:55:07.877373 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 13 15:55:07.877373 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Sep 13 15:55:07.877373 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application Sep 13 15:55:07.877373 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 522/1030] 10.222.0.50 () {56 vars in 1120 bytes} [Tue Sep 13 15:55:07 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 189 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 13 15:55:07.882533 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4448081b-0c83-442b-ad14-fdc2115a9ce8 None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:55:07.883751 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08431c17-8d26-4a14-b46e-9c6679ddf0a5 None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:55:07.921686 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f910761-35f6-49aa-b1a9-964a14b2819f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:55:07.922343 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f910761-35f6-49aa-b1a9-964a14b2819f None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:55:07.922584 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f910761-35f6-49aa-b1a9-964a14b2819f None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:55:07.922821 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f910761-35f6-49aa-b1a9-964a14b2819f None admin] PATH_INFO: `/v3/services` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:55:07.923480 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f910761-35f6-49aa-b1a9-964a14b2819f None admin] RBAC: Adding query filter params (name=image) {{(pid=81438) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:55:07.923780 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f910761-35f6-49aa-b1a9-964a14b2819f None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:55:07.925143 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f910761-35f6-49aa-b1a9-964a14b2819f None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:55:07.931560 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 509/1031] 10.222.0.50 () {56 vars in 1108 bytes} [Tue Sep 13 15:55:07 2022] GET /identity/v3/services?name=image => generated 122 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 13 15:55:07.937146 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f910761-35f6-49aa-b1a9-964a14b2819f None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:55:07.990583 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4448081b-0c83-442b-ad14-fdc2115a9ce8 None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:55:08.028720 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cc27207-24d5-48be-90a6-355218e2cb03 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:55:08.029706 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cc27207-24d5-48be-90a6-355218e2cb03 None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:55:08.030183 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cc27207-24d5-48be-90a6-355218e2cb03 None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:55:08.030656 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cc27207-24d5-48be-90a6-355218e2cb03 None admin] PATH_INFO: `/v3/roles` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:55:08.031543 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2cc27207-24d5-48be-90a6-355218e2cb03 None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=81437) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:55:08.032092 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2cc27207-24d5-48be-90a6-355218e2cb03 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:55:08.033476 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2cc27207-24d5-48be-90a6-355218e2cb03 None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:55:08.046936 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 523/1032] 10.222.0.50 () {56 vars in 1115 bytes} [Tue Sep 13 15:55:07 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 13 15:55:08.048336 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f910761-35f6-49aa-b1a9-964a14b2819f None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:55:08.049758 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cc27207-24d5-48be-90a6-355218e2cb03 None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:55:08.088614 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5845cc08-7065-4944-a677-09af3712c474 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:55:08.089229 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5845cc08-7065-4944-a677-09af3712c474 None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:55:08.089513 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5845cc08-7065-4944-a677-09af3712c474 None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:55:08.089787 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5845cc08-7065-4944-a677-09af3712c474 None admin] PATH_INFO: `/v3/services` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:55:08.090412 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5845cc08-7065-4944-a677-09af3712c474 None admin] RBAC: Adding query filter params (type=image) {{(pid=81438) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:55:08.090719 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5845cc08-7065-4944-a677-09af3712c474 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:55:08.092115 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5845cc08-7065-4944-a677-09af3712c474 None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:55:08.095715 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:55:08.097774 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 510/1033] 10.222.0.50 () {56 vars in 1108 bytes} [Tue Sep 13 15:55:07 2022] GET /identity/v3/services?type=image => generated 350 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 13 15:55:08.104475 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5845cc08-7065-4944-a677-09af3712c474 None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:55:08.109035 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:55:08.159432 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cc27207-24d5-48be-90a6-355218e2cb03 None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:55:08.197764 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56fd152c-697e-4b8f-af92-724cbfd1733c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:55:08.198404 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56fd152c-697e-4b8f-af92-724cbfd1733c None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:55:08.198685 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56fd152c-697e-4b8f-af92-724cbfd1733c None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:55:08.199000 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56fd152c-697e-4b8f-af92-724cbfd1733c None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:55:08.210955 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56fd152c-697e-4b8f-af92-724cbfd1733c None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:55:08.212411 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56fd152c-697e-4b8f-af92-724cbfd1733c None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:55:08.212796 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5845cc08-7065-4944-a677-09af3712c474 None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:55:08.228578 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application [None req-56fd152c-697e-4b8f-af92-724cbfd1733c None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:55:08.228578 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:55:08.228578 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:55:08.228578 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:55:08.228578 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:55:08.228578 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:55:08.228578 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:55:08.228578 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:55:08.228578 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:55:08.228578 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:55:08.228578 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:55:08.228578 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:55:08.228578 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 13 15:55:08.228578 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 13 15:55:08.228578 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 13 15:55:08.228578 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 13 15:55:08.229758 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:55:08.229758 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:55:08.229758 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:55:08.229758 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:55:08.229758 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:55:08.229758 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:55:08.229758 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:55:08.229758 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application with Lock( Sep 13 15:55:08.229758 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:55:08.229758 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:55:08.229758 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:55:08.229758 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:55:08.229758 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:55:08.229758 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:55:08.229758 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:55:08.229758 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:55:08.229758 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 13 15:55:08.230727 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 13 15:55:08.230727 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:55:08.230727 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application Sep 13 15:55:08.230727 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 524/1034] 10.222.0.50 () {56 vars in 1108 bytes} [Tue Sep 13 15:55:08 2022] GET /identity/v3/domains/Default => generated 87 bytes in 181 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 13 15:55:08.236086 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56fd152c-697e-4b8f-af92-724cbfd1733c None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:55:08.252220 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b689795-6a80-4636-8a38-cb8d3333f5f6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:55:08.253051 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b689795-6a80-4636-8a38-cb8d3333f5f6 None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:55:08.253051 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b689795-6a80-4636-8a38-cb8d3333f5f6 None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:55:08.253333 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b689795-6a80-4636-8a38-cb8d3333f5f6 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:55:08.254008 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b689795-6a80-4636-8a38-cb8d3333f5f6 None admin] RBAC: Adding query filter params (interface=public, service_id=4eb6581cfe0444d8b47cf100fb10cb1e, region_id=RegionOne) {{(pid=81438) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:55:08.254345 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b689795-6a80-4636-8a38-cb8d3333f5f6 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:55:08.255799 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b689795-6a80-4636-8a38-cb8d3333f5f6 None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:55:08.261013 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 511/1035] 10.222.0.50 () {56 vars in 1251 bytes} [Tue Sep 13 15:55:08 2022] GET /identity/v3/endpoints?service_id=4eb6581cfe0444d8b47cf100fb10cb1e&interface=public®ion_id=RegionOne => generated 194 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 13 15:55:08.266360 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b689795-6a80-4636-8a38-cb8d3333f5f6 None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:55:08.335353 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:55:08.337612 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:55:08.343144 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56fd152c-697e-4b8f-af92-724cbfd1733c None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:55:08.375562 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b689795-6a80-4636-8a38-cb8d3333f5f6 None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:55:08.377088 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc607adb-f0e1-4dc0-99e9-0fe61850cb91 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:55:08.377694 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc607adb-f0e1-4dc0-99e9-0fe61850cb91 None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:55:08.377971 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc607adb-f0e1-4dc0-99e9-0fe61850cb91 None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:55:08.378242 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc607adb-f0e1-4dc0-99e9-0fe61850cb91 None admin] PATH_INFO: `/v3/domains` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:55:08.378875 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc607adb-f0e1-4dc0-99e9-0fe61850cb91 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81437) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:55:08.379201 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc607adb-f0e1-4dc0-99e9-0fe61850cb91 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:55:08.380561 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc607adb-f0e1-4dc0-99e9-0fe61850cb91 None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:55:08.393108 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 525/1036] 10.222.0.50 () {56 vars in 1109 bytes} [Tue Sep 13 15:55:08 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 13 15:55:08.398439 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc607adb-f0e1-4dc0-99e9-0fe61850cb91 None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:55:08.411504 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28ce9569-4a6b-4bf7-a50c-f0d460ee8111 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:55:08.412117 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28ce9569-4a6b-4bf7-a50c-f0d460ee8111 None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:55:08.412398 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28ce9569-4a6b-4bf7-a50c-f0d460ee8111 None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:55:08.413252 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28ce9569-4a6b-4bf7-a50c-f0d460ee8111 None admin] PATH_INFO: `/v3/services` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:55:08.413854 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28ce9569-4a6b-4bf7-a50c-f0d460ee8111 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:55:08.415262 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28ce9569-4a6b-4bf7-a50c-f0d460ee8111 None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:55:08.421919 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 512/1037] 10.222.0.50 () {56 vars in 1087 bytes} [Tue Sep 13 15:55:08 2022] GET /identity/v3/services => generated 1957 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 13 15:55:08.507805 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc607adb-f0e1-4dc0-99e9-0fe61850cb91 None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:55:08.547391 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfebfdcf-de25-4b95-8832-c142b669db6d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:55:08.548116 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfebfdcf-de25-4b95-8832-c142b669db6d None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:55:08.548460 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfebfdcf-de25-4b95-8832-c142b669db6d None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:55:08.548796 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfebfdcf-de25-4b95-8832-c142b669db6d None admin] PATH_INFO: `/v3/users/nova` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:55:08.557361 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dfebfdcf-de25-4b95-8832-c142b669db6d None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:55:08.558923 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dfebfdcf-de25-4b95-8832-c142b669db6d None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:55:08.571540 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application [None req-dfebfdcf-de25-4b95-8832-c142b669db6d None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Sep 13 15:55:08.571540 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:55:08.571540 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:55:08.571540 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:55:08.571540 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:55:08.571540 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:55:08.571540 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:55:08.571540 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:55:08.571540 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:55:08.571540 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:55:08.571540 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:55:08.571540 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:55:08.571540 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 13 15:55:08.571540 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 13 15:55:08.571540 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 13 15:55:08.571540 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 13 15:55:08.572787 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:55:08.572787 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:55:08.572787 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 13 15:55:08.572787 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 13 15:55:08.572787 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 13 15:55:08.572787 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 13 15:55:08.572787 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:55:08.572787 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:55:08.572787 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:55:08.572787 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:55:08.572787 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:55:08.572787 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application with Lock( Sep 13 15:55:08.572787 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:55:08.572787 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:55:08.572787 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:55:08.572787 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:55:08.572787 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:55:08.572787 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:55:08.573986 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:55:08.573986 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:55:08.573986 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 13 15:55:08.573986 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 13 15:55:08.573986 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 13 15:55:08.573986 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 13 15:55:08.573986 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 13 15:55:08.573986 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 13 15:55:08.573986 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Sep 13 15:55:08.573986 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application Sep 13 15:55:08.575373 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 526/1038] 10.222.0.50 () {56 vars in 1093 bytes} [Tue Sep 13 15:55:08 2022] GET /identity/v3/users/nova => generated 82 bytes in 177 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 13 15:55:08.579979 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28ce9569-4a6b-4bf7-a50c-f0d460ee8111 None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:55:08.682680 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28ce9569-4a6b-4bf7-a50c-f0d460ee8111 None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:55:08.711283 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:55:08.717459 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2f08acf-cada-4b3a-883c-c71e3797b5ea None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:55:08.718064 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2f08acf-cada-4b3a-883c-c71e3797b5ea None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:55:08.718343 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2f08acf-cada-4b3a-883c-c71e3797b5ea None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:55:08.718609 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2f08acf-cada-4b3a-883c-c71e3797b5ea None admin] PATH_INFO: `/v3/users` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:55:08.719369 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2f08acf-cada-4b3a-883c-c71e3797b5ea None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81438) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:55:08.719671 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2f08acf-cada-4b3a-883c-c71e3797b5ea None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:55:08.721072 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2f08acf-cada-4b3a-883c-c71e3797b5ea None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:55:08.738849 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 513/1039] 10.222.0.50 () {56 vars in 1133 bytes} [Tue Sep 13 15:55:08 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:55:08.744341 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfebfdcf-de25-4b95-8832-c142b669db6d None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:55:08.760421 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:55:08.846741 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfebfdcf-de25-4b95-8832-c142b669db6d None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:55:08.889830 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-266ff11e-e7ee-4171-87bb-cea8541748dd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:55:08.890494 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-266ff11e-e7ee-4171-87bb-cea8541748dd None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:55:08.890726 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-266ff11e-e7ee-4171-87bb-cea8541748dd None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:55:08.890989 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-266ff11e-e7ee-4171-87bb-cea8541748dd None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:55:08.960312 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-266ff11e-e7ee-4171-87bb-cea8541748dd None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:55:08.961762 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-266ff11e-e7ee-4171-87bb-cea8541748dd None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:55:08.977231 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:55:09.067845 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 527/1040] 10.222.0.50 () {58 vars in 1282 bytes} [Tue Sep 13 15:55:08 2022] GET /identity/v3/auth/tokens => generated 1969 bytes in 324 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 13 15:55:09.073596 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2f08acf-cada-4b3a-883c-c71e3797b5ea None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:55:09.176514 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2f08acf-cada-4b3a-883c-c71e3797b5ea None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:55:09.212404 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8877c344-7530-4c89-9730-7321fe5ae655 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:55:09.213107 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8877c344-7530-4c89-9730-7321fe5ae655 None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:55:09.213393 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8877c344-7530-4c89-9730-7321fe5ae655 None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:55:09.213669 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8877c344-7530-4c89-9730-7321fe5ae655 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:55:09.223993 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8877c344-7530-4c89-9730-7321fe5ae655 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:55:09.226009 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8877c344-7530-4c89-9730-7321fe5ae655 None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:55:09.240775 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application [None req-8877c344-7530-4c89-9730-7321fe5ae655 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:55:09.240775 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:55:09.240775 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:55:09.240775 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:55:09.240775 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:55:09.240775 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:55:09.240775 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:55:09.240775 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:55:09.240775 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:55:09.240775 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:55:09.240775 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:55:09.240775 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:55:09.240775 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 13 15:55:09.240775 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 13 15:55:09.240775 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 13 15:55:09.240775 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 13 15:55:09.241845 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:55:09.241845 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:55:09.241845 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:55:09.241845 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:55:09.241845 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:55:09.241845 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:55:09.241845 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:55:09.241845 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application with Lock( Sep 13 15:55:09.241845 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:55:09.241845 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:55:09.241845 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:55:09.241845 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:55:09.241845 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:55:09.241845 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:55:09.241845 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:55:09.241845 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:55:09.241845 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 13 15:55:09.247326 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 13 15:55:09.247326 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:55:09.247326 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Sep 13 15:55:09.247326 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 514/1041] 10.222.0.50 () {56 vars in 1108 bytes} [Tue Sep 13 15:55:09 2022] GET /identity/v3/domains/Default => generated 87 bytes in 169 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 13 15:55:09.247655 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-266ff11e-e7ee-4171-87bb-cea8541748dd None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:55:09.352646 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-266ff11e-e7ee-4171-87bb-cea8541748dd None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:55:09.367865 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:55:09.389485 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a147fc1-450e-401c-830a-e8355416441a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:55:09.390096 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a147fc1-450e-401c-830a-e8355416441a None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:55:09.390375 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a147fc1-450e-401c-830a-e8355416441a None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:55:09.390648 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a147fc1-450e-401c-830a-e8355416441a None admin] PATH_INFO: `/v3/domains` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:55:09.391319 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a147fc1-450e-401c-830a-e8355416441a None admin] RBAC: Adding query filter params (name=Default) {{(pid=81437) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:55:09.391622 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a147fc1-450e-401c-830a-e8355416441a None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:55:09.392983 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a147fc1-450e-401c-830a-e8355416441a None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:55:09.408869 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 528/1042] 10.222.0.50 () {56 vars in 1109 bytes} [Tue Sep 13 15:55:09 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:55:09.410152 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8877c344-7530-4c89-9730-7321fe5ae655 None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:55:09.456839 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:55:09.515508 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8877c344-7530-4c89-9730-7321fe5ae655 None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:55:09.549051 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04127496-79cf-4603-ad37-959678b720bf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:55:09.549679 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04127496-79cf-4603-ad37-959678b720bf None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:55:09.549952 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04127496-79cf-4603-ad37-959678b720bf None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:55:09.550219 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04127496-79cf-4603-ad37-959678b720bf None admin] PATH_INFO: `/v3/projects/service` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:55:09.557972 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04127496-79cf-4603-ad37-959678b720bf None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:55:09.559436 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04127496-79cf-4603-ad37-959678b720bf None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:55:09.571944 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application [None req-04127496-79cf-4603-ad37-959678b720bf None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 13 15:55:09.571944 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:55:09.571944 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:55:09.571944 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:55:09.571944 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:55:09.571944 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:55:09.571944 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:55:09.571944 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:55:09.571944 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:55:09.571944 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:55:09.571944 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:55:09.571944 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:55:09.571944 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 13 15:55:09.571944 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 13 15:55:09.571944 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 13 15:55:09.572882 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 13 15:55:09.572882 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:55:09.572882 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:55:09.572882 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:55:09.572882 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:55:09.572882 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:55:09.572882 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:55:09.572882 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:55:09.572882 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application with Lock( Sep 13 15:55:09.572882 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:55:09.572882 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:55:09.572882 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:55:09.572882 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:55:09.572882 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:55:09.572882 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:55:09.572882 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:55:09.572882 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:55:09.573806 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 13 15:55:09.573806 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 13 15:55:09.573806 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 13 15:55:09.573806 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 13 15:55:09.573806 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 13 15:55:09.573806 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 13 15:55:09.573806 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 13 15:55:09.573806 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Sep 13 15:55:09.577960 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 515/1043] 10.222.0.50 () {56 vars in 1111 bytes} [Tue Sep 13 15:55:09 2022] GET /identity/v3/projects/service => generated 88 bytes in 168 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:55:09.579407 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a147fc1-450e-401c-830a-e8355416441a None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:55:09.689230 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a147fc1-450e-401c-830a-e8355416441a None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:55:09.725970 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eec95240-4670-4873-9d22-962d13ddafb1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:55:09.726607 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eec95240-4670-4873-9d22-962d13ddafb1 None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:55:09.726885 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eec95240-4670-4873-9d22-962d13ddafb1 None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:55:09.727187 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eec95240-4670-4873-9d22-962d13ddafb1 None admin] PATH_INFO: `/v3/projects` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:55:09.727896 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eec95240-4670-4873-9d22-962d13ddafb1 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81437) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:55:09.728205 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eec95240-4670-4873-9d22-962d13ddafb1 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:55:09.729705 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eec95240-4670-4873-9d22-962d13ddafb1 None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:55:09.748969 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 529/1044] 10.222.0.50 () {56 vars in 1148 bytes} [Tue Sep 13 15:55:09 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:55:09.750611 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04127496-79cf-4603-ad37-959678b720bf None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:55:09.835902 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b2462ec-02fb-435f-a933-497eb66199f5 None None] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:55:09.836241 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b2462ec-02fb-435f-a933-497eb66199f5 None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:55:09.836456 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b2462ec-02fb-435f-a933-497eb66199f5 None None] PATH_INFO: `/` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:55:09.837958 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 530/1045] 10.222.0.50 () {54 vars in 927 bytes} [Tue Sep 13 15:55:09 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 13 15:55:09.842681 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43321037-0412-4f99-a166-6eaabd538f94 None None] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:55:09.843026 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43321037-0412-4f99-a166-6eaabd538f94 None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:55:09.843283 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43321037-0412-4f99-a166-6eaabd538f94 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:55:09.853831 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04127496-79cf-4603-ad37-959678b720bf None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:55:09.855416 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:55:09.861764 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.auth.core [None req-43321037-0412-4f99-a166-6eaabd538f94 None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81437) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:55:09.862993 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-43321037-0412-4f99-a166-6eaabd538f94 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:55:09.891231 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:55:09.891823 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfe1e971-ed2c-4f8f-9e7a-e4ddd3fb681a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:55:09.892251 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfe1e971-ed2c-4f8f-9e7a-e4ddd3fb681a None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:55:09.892565 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfe1e971-ed2c-4f8f-9e7a-e4ddd3fb681a None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:55:09.892851 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfe1e971-ed2c-4f8f-9e7a-e4ddd3fb681a None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:55:09.893650 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dfe1e971-ed2c-4f8f-9e7a-e4ddd3fb681a None admin] RBAC: Adding query filter params (role.id=20e2f11234344900a51fc76fe5b625f3, scope.project.id=5811e3b83a1e462e9320cad79a3ad791, user.id=4e039b46c5e3445bbb39904b00c5430e) {{(pid=81438) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:55:09.893994 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dfe1e971-ed2c-4f8f-9e7a-e4ddd3fb681a None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:55:09.895476 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dfe1e971-ed2c-4f8f-9e7a-e4ddd3fb681a None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:55:09.901128 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 516/1046] 10.222.0.50 () {56 vars in 1374 bytes} [Tue Sep 13 15:55:09 2022] GET /identity/v3/role_assignments?user.id=4e039b46c5e3445bbb39904b00c5430e&scope.project.id=5811e3b83a1e462e9320cad79a3ad791&role.id=20e2f11234344900a51fc76fe5b625f3 => generated 259 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:55:09.950310 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 531/1047] 10.222.0.50 () {58 vars in 1024 bytes} [Tue Sep 13 15:55:09 2022] POST /identity/v3/auth/tokens => generated 1969 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 13 15:55:10.225495 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfe1e971-ed2c-4f8f-9e7a-e4ddd3fb681a None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:55:10.326974 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfe1e971-ed2c-4f8f-9e7a-e4ddd3fb681a None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:55:10.360953 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf15d6bf-4ea3-43e1-90f9-5b337b6acb0a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:55:10.361587 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf15d6bf-4ea3-43e1-90f9-5b337b6acb0a None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:55:10.361863 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf15d6bf-4ea3-43e1-90f9-5b337b6acb0a None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:55:10.362135 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf15d6bf-4ea3-43e1-90f9-5b337b6acb0a None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:55:10.374195 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf15d6bf-4ea3-43e1-90f9-5b337b6acb0a None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 13 15:55:10.378985 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application [None req-bf15d6bf-4ea3-43e1-90f9-5b337b6acb0a None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 13 15:55:10.378985 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:55:10.378985 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:55:10.378985 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:55:10.378985 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:55:10.378985 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:55:10.378985 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:55:10.378985 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:55:10.378985 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:55:10.378985 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:55:10.378985 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:55:10.378985 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:55:10.378985 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 13 15:55:10.378985 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 13 15:55:10.378985 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 13 15:55:10.378985 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 13 15:55:10.379956 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 13 15:55:10.379956 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 13 15:55:10.379956 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 13 15:55:10.379956 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 13 15:55:10.379956 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:55:10.379956 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:55:10.379956 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:55:10.379956 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:55:10.379956 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:55:10.379956 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:55:10.379956 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:55:10.379956 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application with Lock( Sep 13 15:55:10.379956 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:55:10.379956 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:55:10.379956 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:55:10.379956 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:55:10.383943 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:55:10.383943 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:55:10.383943 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:55:10.383943 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:55:10.383943 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 13 15:55:10.383943 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 13 15:55:10.383943 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 13 15:55:10.383943 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 13 15:55:10.383943 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 13 15:55:10.383943 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 13 15:55:10.383943 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 13 15:55:10.383943 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Sep 13 15:55:10.383943 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 517/1048] 10.222.0.50 () {56 vars in 1096 bytes} [Tue Sep 13 15:55:10 2022] GET /identity/v3/roles/admin => generated 83 bytes in 156 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:55:10.386514 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43321037-0412-4f99-a166-6eaabd538f94 None None] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:55:10.489362 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43321037-0412-4f99-a166-6eaabd538f94 None None] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:55:10.527969 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-773b7562-9b93-4de8-8cdc-0c60b6e42667 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:55:10.528586 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-773b7562-9b93-4de8-8cdc-0c60b6e42667 None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:55:10.528865 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-773b7562-9b93-4de8-8cdc-0c60b6e42667 None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:55:10.529138 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-773b7562-9b93-4de8-8cdc-0c60b6e42667 None admin] PATH_INFO: `/v3/roles` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:55:10.529801 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-773b7562-9b93-4de8-8cdc-0c60b6e42667 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81437) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:55:10.530102 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-773b7562-9b93-4de8-8cdc-0c60b6e42667 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:55:10.531523 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-773b7562-9b93-4de8-8cdc-0c60b6e42667 None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:55:10.540537 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 532/1049] 10.222.0.50 () {56 vars in 1099 bytes} [Tue Sep 13 15:55:10 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:55:10.546358 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf15d6bf-4ea3-43e1-90f9-5b337b6acb0a None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:55:10.575455 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:55:10.610599 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-220f43d1-4c2a-4bea-91d4-77946cc9cad4 None None] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:55:10.610887 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-220f43d1-4c2a-4bea-91d4-77946cc9cad4 None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:55:10.611196 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-220f43d1-4c2a-4bea-91d4-77946cc9cad4 None None] PATH_INFO: `/` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:55:10.612776 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 533/1050] 10.222.0.50 () {54 vars in 927 bytes} [Tue Sep 13 15:55:10 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:55:10.618343 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78253d59-24c1-4094-8c12-fbbcd1e75a6d None None] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:55:10.618616 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78253d59-24c1-4094-8c12-fbbcd1e75a6d None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:55:10.618912 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78253d59-24c1-4094-8c12-fbbcd1e75a6d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:55:10.637242 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.auth.core [None req-78253d59-24c1-4094-8c12-fbbcd1e75a6d None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81437) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:55:10.638413 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-78253d59-24c1-4094-8c12-fbbcd1e75a6d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:55:10.654262 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf15d6bf-4ea3-43e1-90f9-5b337b6acb0a None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:55:10.659982 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:55:10.690413 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a6f5d4d-689c-4891-9769-5ebe4632063c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:55:10.691092 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a6f5d4d-689c-4891-9769-5ebe4632063c None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:55:10.691377 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a6f5d4d-689c-4891-9769-5ebe4632063c None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:55:10.691660 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a6f5d4d-689c-4891-9769-5ebe4632063c None admin] PATH_INFO: `/v3/users/4d24fa39b5ff4de7a57641b0bf2c1044` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:55:10.705843 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a6f5d4d-689c-4891-9769-5ebe4632063c None admin] RBAC: Authorizing `identity:get_user(user_id=4d24fa39b5ff4de7a57641b0bf2c1044)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:55:10.707349 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a6f5d4d-689c-4891-9769-5ebe4632063c None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:55:10.710934 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 518/1051] 10.222.0.50 () {56 vars in 1177 bytes} [Tue Sep 13 15:55:10 2022] GET /identity/v3/users/4d24fa39b5ff4de7a57641b0bf2c1044 => generated 285 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:55:10.715684 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a6f5d4d-689c-4891-9769-5ebe4632063c None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:55:10.719180 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 534/1052] 10.222.0.50 () {58 vars in 1024 bytes} [Tue Sep 13 15:55:10 2022] POST /identity/v3/auth/tokens => generated 1969 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 13 15:55:10.802079 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:55:10.819979 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a6f5d4d-689c-4891-9769-5ebe4632063c None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:55:10.855895 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39ec55b8-00cb-4a3f-8220-ac5523f634b8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:55:10.856499 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39ec55b8-00cb-4a3f-8220-ac5523f634b8 None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:55:10.856778 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39ec55b8-00cb-4a3f-8220-ac5523f634b8 None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:55:10.857047 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39ec55b8-00cb-4a3f-8220-ac5523f634b8 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:55:10.926292 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39ec55b8-00cb-4a3f-8220-ac5523f634b8 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:55:10.927751 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39ec55b8-00cb-4a3f-8220-ac5523f634b8 None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:55:10.988291 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78253d59-24c1-4094-8c12-fbbcd1e75a6d None None] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:55:11.016813 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:55:11.034937 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 519/1053] 10.222.0.50 () {58 vars in 1282 bytes} [Tue Sep 13 15:55:10 2022] GET /identity/v3/auth/tokens => generated 1969 bytes in 320 msecs (HTTP/1.1 200) 6 headers in 359 bytes (2 switches on core 0) Sep 13 15:55:11.043298 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39ec55b8-00cb-4a3f-8220-ac5523f634b8 None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:55:11.108265 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78253d59-24c1-4094-8c12-fbbcd1e75a6d None None] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:55:11.144350 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f76dd792-3724-43be-8dcb-7bc45c8cffbf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:55:11.144990 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f76dd792-3724-43be-8dcb-7bc45c8cffbf None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:55:11.145271 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f76dd792-3724-43be-8dcb-7bc45c8cffbf None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:55:11.145539 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f76dd792-3724-43be-8dcb-7bc45c8cffbf None admin] PATH_INFO: `/v3/services/image` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:55:11.146122 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f76dd792-3724-43be-8dcb-7bc45c8cffbf None admin] RBAC: Authorizing `identity:get_service(service_id=image)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:55:11.148388 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f76dd792-3724-43be-8dcb-7bc45c8cffbf None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:55:11.154708 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39ec55b8-00cb-4a3f-8220-ac5523f634b8 None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:55:11.157178 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application [None req-f76dd792-3724-43be-8dcb-7bc45c8cffbf None admin] Could not find service: image.: keystone.exception.ServiceNotFound: Could not find service: image. Sep 13 15:55:11.157178 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:55:11.157178 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 13 15:55:11.157178 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 13 15:55:11.157178 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 13 15:55:11.157178 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 13 15:55:11.157178 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 13 15:55:11.157178 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 13 15:55:11.157178 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Sep 13 15:55:11.157178 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application Sep 13 15:55:11.157178 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 13 15:55:11.157178 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application Sep 13 15:55:11.157178 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:55:11.157178 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:55:11.157178 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:55:11.157178 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:55:11.157178 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:55:11.158289 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:55:11.158289 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:55:11.158289 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:55:11.158289 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:55:11.158289 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:55:11.158289 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:55:11.158289 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 13 15:55:11.158289 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 13 15:55:11.158289 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 13 15:55:11.158289 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 13 15:55:11.158289 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:55:11.158289 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:55:11.158289 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:55:11.158289 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:55:11.158289 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:55:11.158289 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:55:11.159721 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:55:11.159721 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application with Lock( Sep 13 15:55:11.159721 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:55:11.159721 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:55:11.159721 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:55:11.159721 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:55:11.159721 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:55:11.159721 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:55:11.159721 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:55:11.159721 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:55:11.159721 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 13 15:55:11.159721 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 13 15:55:11.159721 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Sep 13 15:55:11.159721 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application Sep 13 15:55:11.159721 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 535/1054] 10.222.0.50 () {56 vars in 1105 bytes} [Tue Sep 13 15:55:10 2022] GET /identity/v3/services/image => generated 86 bytes in 170 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 13 15:55:11.164150 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f76dd792-3724-43be-8dcb-7bc45c8cffbf None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:55:11.186294 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:55:11.196874 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8415f779-53bf-4ac9-99ff-5d444478419b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:55:11.197579 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8415f779-53bf-4ac9-99ff-5d444478419b None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:55:11.197927 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8415f779-53bf-4ac9-99ff-5d444478419b None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:55:11.198209 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8415f779-53bf-4ac9-99ff-5d444478419b None admin] PATH_INFO: `/v3/projects/13d63282b1fd48349397eed6fccbd62a` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:55:11.199456 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8415f779-53bf-4ac9-99ff-5d444478419b None admin] RBAC: Authorizing `identity:get_project(project_id=13d63282b1fd48349397eed6fccbd62a)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:55:11.200886 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8415f779-53bf-4ac9-99ff-5d444478419b None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:55:11.202686 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 520/1055] 10.222.0.50 () {56 vars in 1186 bytes} [Tue Sep 13 15:55:11 2022] GET /identity/v3/projects/13d63282b1fd48349397eed6fccbd62a => generated 311 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:55:11.208313 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8415f779-53bf-4ac9-99ff-5d444478419b None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:55:11.275312 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f76dd792-3724-43be-8dcb-7bc45c8cffbf None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:55:11.275930 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:55:11.316564 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d36cd7e-d10e-477b-b58f-6dff21122058 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:55:11.317094 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d36cd7e-d10e-477b-b58f-6dff21122058 None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:55:11.317380 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d36cd7e-d10e-477b-b58f-6dff21122058 None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:55:11.317655 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d36cd7e-d10e-477b-b58f-6dff21122058 None admin] PATH_INFO: `/v3/services` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:55:11.318276 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d36cd7e-d10e-477b-b58f-6dff21122058 None admin] RBAC: Adding query filter params (name=image) {{(pid=81437) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:55:11.318580 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d36cd7e-d10e-477b-b58f-6dff21122058 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:55:11.320503 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d36cd7e-d10e-477b-b58f-6dff21122058 None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:55:11.325721 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 536/1056] 10.222.0.50 () {56 vars in 1108 bytes} [Tue Sep 13 15:55:11 2022] GET /identity/v3/services?name=image => generated 122 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 13 15:55:11.328844 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8415f779-53bf-4ac9-99ff-5d444478419b None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:55:11.330958 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d36cd7e-d10e-477b-b58f-6dff21122058 None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:55:11.365185 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6853fc65-e35d-4933-a969-72a1259d26a9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:55:11.366383 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6853fc65-e35d-4933-a969-72a1259d26a9 None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:55:11.366840 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6853fc65-e35d-4933-a969-72a1259d26a9 None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:55:11.367335 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6853fc65-e35d-4933-a969-72a1259d26a9 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:55:11.368053 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6853fc65-e35d-4933-a969-72a1259d26a9 None admin] RBAC: Adding query filter params (role.id=031d4e1a23e34f71858fc4a6dfbd4ccb, scope.project.id=13d63282b1fd48349397eed6fccbd62a, user.id=4d24fa39b5ff4de7a57641b0bf2c1044) {{(pid=81438) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:55:11.368361 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6853fc65-e35d-4933-a969-72a1259d26a9 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:55:11.369838 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6853fc65-e35d-4933-a969-72a1259d26a9 None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:55:11.383307 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 521/1057] 10.222.0.50 () {56 vars in 1374 bytes} [Tue Sep 13 15:55:11 2022] GET /identity/v3/role_assignments?user.id=4d24fa39b5ff4de7a57641b0bf2c1044&scope.project.id=13d63282b1fd48349397eed6fccbd62a&role.id=031d4e1a23e34f71858fc4a6dfbd4ccb => generated 259 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 13 15:55:11.423382 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:55:11.443218 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d36cd7e-d10e-477b-b58f-6dff21122058 None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:55:11.481359 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-627b6d58-8374-40ac-b4ec-3b1106709ff8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:55:11.481979 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-627b6d58-8374-40ac-b4ec-3b1106709ff8 None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:55:11.482264 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-627b6d58-8374-40ac-b4ec-3b1106709ff8 None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:55:11.482537 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-627b6d58-8374-40ac-b4ec-3b1106709ff8 None admin] PATH_INFO: `/v3/services` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:55:11.483212 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-627b6d58-8374-40ac-b4ec-3b1106709ff8 None admin] RBAC: Adding query filter params (type=image) {{(pid=81437) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:55:11.483518 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-627b6d58-8374-40ac-b4ec-3b1106709ff8 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:55:11.484908 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-627b6d58-8374-40ac-b4ec-3b1106709ff8 None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:55:11.495338 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 537/1058] 10.222.0.50 () {56 vars in 1108 bytes} [Tue Sep 13 15:55:11 2022] GET /identity/v3/services?type=image => generated 350 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 13 15:55:11.496709 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6853fc65-e35d-4933-a969-72a1259d26a9 None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:55:11.607527 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6853fc65-e35d-4933-a969-72a1259d26a9 None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:55:11.624728 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:55:11.644223 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26d191a7-79c9-4c9f-82ed-e1ccb8eaa590 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:55:11.644841 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26d191a7-79c9-4c9f-82ed-e1ccb8eaa590 None admin] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:55:11.645122 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26d191a7-79c9-4c9f-82ed-e1ccb8eaa590 None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:55:11.645395 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26d191a7-79c9-4c9f-82ed-e1ccb8eaa590 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:55:11.646059 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26d191a7-79c9-4c9f-82ed-e1ccb8eaa590 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:55:11.648050 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26d191a7-79c9-4c9f-82ed-e1ccb8eaa590 None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:55:11.674936 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 522/1059] 10.222.0.50 () {60 vars in 1144 bytes} [Tue Sep 13 15:55:11 2022] POST /identity/v3/endpoints => generated 327 bytes in 174 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 15:55:12.060686 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57c7a6e3-fdf1-44ff-bfe1-8d94ffda2093 None None] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:55:12.060686 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57c7a6e3-fdf1-44ff-bfe1-8d94ffda2093 None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:55:12.060686 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57c7a6e3-fdf1-44ff-bfe1-8d94ffda2093 None None] PATH_INFO: `/` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:55:12.061452 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 538/1060] 10.222.0.50 () {54 vars in 927 bytes} [Tue Sep 13 15:55:12 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:55:12.066724 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00b28296-9a76-4802-bca6-2eb16ef62bc2 None None] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:55:12.067057 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00b28296-9a76-4802-bca6-2eb16ef62bc2 None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:55:12.067333 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00b28296-9a76-4802-bca6-2eb16ef62bc2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:55:12.085781 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.auth.core [None req-00b28296-9a76-4802-bca6-2eb16ef62bc2 None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81438) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:55:12.087005 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-00b28296-9a76-4802-bca6-2eb16ef62bc2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:55:12.160105 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 523/1061] 10.222.0.50 () {58 vars in 1024 bytes} [Tue Sep 13 15:55:12 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 13 15:55:12.428449 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57c7a6e3-fdf1-44ff-bfe1-8d94ffda2093 None None] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:55:12.530108 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57c7a6e3-fdf1-44ff-bfe1-8d94ffda2093 None None] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:55:12.564859 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efc8adff-a0a1-4b92-b6a3-452caf1acfe7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:55:12.565508 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efc8adff-a0a1-4b92-b6a3-452caf1acfe7 None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:55:12.565785 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efc8adff-a0a1-4b92-b6a3-452caf1acfe7 None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:55:12.566055 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efc8adff-a0a1-4b92-b6a3-452caf1acfe7 None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:55:12.578982 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efc8adff-a0a1-4b92-b6a3-452caf1acfe7 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 13 15:55:12.583865 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application [None req-efc8adff-a0a1-4b92-b6a3-452caf1acfe7 None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Sep 13 15:55:12.583865 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:55:12.583865 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:55:12.583865 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:55:12.583865 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:55:12.583865 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:55:12.583865 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:55:12.583865 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:55:12.583865 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:55:12.583865 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:55:12.583865 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:55:12.583865 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:55:12.583865 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 13 15:55:12.583865 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 13 15:55:12.583865 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 13 15:55:12.583865 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 13 15:55:12.585569 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 13 15:55:12.585569 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 13 15:55:12.585569 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 13 15:55:12.585569 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 13 15:55:12.585569 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:55:12.585569 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:55:12.585569 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:55:12.585569 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:55:12.585569 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:55:12.585569 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:55:12.585569 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:55:12.585569 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application with Lock( Sep 13 15:55:12.585569 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:55:12.585569 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:55:12.585569 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:55:12.585569 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:55:12.586426 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:55:12.586426 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:55:12.586426 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:55:12.586426 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:55:12.586426 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 13 15:55:12.586426 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 13 15:55:12.586426 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 13 15:55:12.586426 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 13 15:55:12.586426 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 13 15:55:12.586426 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 13 15:55:12.586426 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Sep 13 15:55:12.586426 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application Sep 13 15:55:12.586426 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 539/1062] 10.222.0.50 () {56 vars in 1120 bytes} [Tue Sep 13 15:55:12 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 158 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:55:12.591947 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00b28296-9a76-4802-bca6-2eb16ef62bc2 None None] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:55:12.690942 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:55:12.695718 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00b28296-9a76-4802-bca6-2eb16ef62bc2 None None] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:55:12.731238 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59f96e11-6d1b-43a8-beba-daf4f8c248be None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:55:12.731849 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59f96e11-6d1b-43a8-beba-daf4f8c248be None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:55:12.732129 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59f96e11-6d1b-43a8-beba-daf4f8c248be None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:55:12.732404 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59f96e11-6d1b-43a8-beba-daf4f8c248be None admin] PATH_INFO: `/v3/roles` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:55:12.733056 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59f96e11-6d1b-43a8-beba-daf4f8c248be None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=81438) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:55:12.733358 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59f96e11-6d1b-43a8-beba-daf4f8c248be None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:55:12.734718 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59f96e11-6d1b-43a8-beba-daf4f8c248be None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:55:12.747193 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 524/1063] 10.222.0.50 () {56 vars in 1115 bytes} [Tue Sep 13 15:55:12 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:55:12.748373 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efc8adff-a0a1-4b92-b6a3-452caf1acfe7 None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:55:12.760391 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:55:12.850398 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efc8adff-a0a1-4b92-b6a3-452caf1acfe7 None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:55:12.884380 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f4f1f8b-d4c3-463b-90c1-e029e468ba96 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:55:12.885009 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f4f1f8b-d4c3-463b-90c1-e029e468ba96 None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:55:12.885287 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f4f1f8b-d4c3-463b-90c1-e029e468ba96 None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:55:12.885559 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f4f1f8b-d4c3-463b-90c1-e029e468ba96 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:55:12.896833 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f4f1f8b-d4c3-463b-90c1-e029e468ba96 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:55:12.898802 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f4f1f8b-d4c3-463b-90c1-e029e468ba96 None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:55:12.914348 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application [None req-8f4f1f8b-d4c3-463b-90c1-e029e468ba96 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:55:12.914348 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:55:12.914348 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:55:12.914348 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:55:12.914348 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:55:12.914348 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:55:12.914348 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:55:12.914348 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:55:12.914348 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:55:12.914348 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:55:12.914348 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:55:12.914348 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:55:12.914348 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 13 15:55:12.914348 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 13 15:55:12.914348 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 13 15:55:12.914348 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 13 15:55:12.915521 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:55:12.915521 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:55:12.915521 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:55:12.915521 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:55:12.915521 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:55:12.915521 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:55:12.915521 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:55:12.915521 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application with Lock( Sep 13 15:55:12.915521 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:55:12.915521 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:55:12.915521 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:55:12.915521 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:55:12.915521 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:55:12.915521 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:55:12.915521 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:55:12.915521 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:55:12.915521 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 13 15:55:12.916449 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 13 15:55:12.916449 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:55:12.916449 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application Sep 13 15:55:12.916449 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 540/1064] 10.222.0.50 () {56 vars in 1108 bytes} [Tue Sep 13 15:55:12 2022] GET /identity/v3/domains/Default => generated 87 bytes in 168 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 13 15:55:12.922936 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59f96e11-6d1b-43a8-beba-daf4f8c248be None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:55:13.025387 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59f96e11-6d1b-43a8-beba-daf4f8c248be None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:55:13.067948 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0274cc9-6a80-4039-9896-af5ecdd781b3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:55:13.069182 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0274cc9-6a80-4039-9896-af5ecdd781b3 None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:55:13.069469 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0274cc9-6a80-4039-9896-af5ecdd781b3 None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:55:13.069763 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0274cc9-6a80-4039-9896-af5ecdd781b3 None admin] PATH_INFO: `/v3/domains` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:55:13.070385 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0274cc9-6a80-4039-9896-af5ecdd781b3 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81438) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:55:13.070688 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0274cc9-6a80-4039-9896-af5ecdd781b3 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:55:13.072083 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0274cc9-6a80-4039-9896-af5ecdd781b3 None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:55:13.084120 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 525/1065] 10.222.0.50 () {56 vars in 1109 bytes} [Tue Sep 13 15:55:12 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:55:13.090455 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f4f1f8b-d4c3-463b-90c1-e029e468ba96 None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:55:13.152564 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:55:13.196291 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f4f1f8b-d4c3-463b-90c1-e029e468ba96 None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:55:13.230454 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0dffefcc-520d-45c7-9388-f528f5d2ec9f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:55:13.231113 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dffefcc-520d-45c7-9388-f528f5d2ec9f None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:55:13.231399 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dffefcc-520d-45c7-9388-f528f5d2ec9f None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:55:13.231675 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dffefcc-520d-45c7-9388-f528f5d2ec9f None admin] PATH_INFO: `/v3/users/nova` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:55:13.239286 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0dffefcc-520d-45c7-9388-f528f5d2ec9f None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:55:13.241385 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0dffefcc-520d-45c7-9388-f528f5d2ec9f None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:55:13.252851 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application [None req-0dffefcc-520d-45c7-9388-f528f5d2ec9f None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Sep 13 15:55:13.252851 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:55:13.252851 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:55:13.252851 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:55:13.252851 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:55:13.252851 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:55:13.252851 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:55:13.252851 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:55:13.252851 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:55:13.252851 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:55:13.252851 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:55:13.252851 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:55:13.252851 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 13 15:55:13.252851 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 13 15:55:13.252851 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 13 15:55:13.252851 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 13 15:55:13.253823 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:55:13.253823 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:55:13.253823 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 13 15:55:13.253823 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 13 15:55:13.253823 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 13 15:55:13.253823 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 13 15:55:13.253823 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:55:13.253823 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:55:13.253823 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:55:13.253823 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:55:13.253823 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:55:13.253823 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application with Lock( Sep 13 15:55:13.253823 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:55:13.253823 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:55:13.253823 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:55:13.253823 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:55:13.253823 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:55:13.253823 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:55:13.254784 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:55:13.254784 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:55:13.254784 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 13 15:55:13.254784 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 13 15:55:13.254784 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 13 15:55:13.254784 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 13 15:55:13.254784 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 13 15:55:13.254784 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 13 15:55:13.254784 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Sep 13 15:55:13.254784 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application Sep 13 15:55:13.254784 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 541/1066] 10.222.0.50 () {56 vars in 1093 bytes} [Tue Sep 13 15:55:13 2022] GET /identity/v3/users/nova => generated 82 bytes in 164 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 13 15:55:13.260380 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0274cc9-6a80-4039-9896-af5ecdd781b3 None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:55:13.283422 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:55:13.365264 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0274cc9-6a80-4039-9896-af5ecdd781b3 None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:55:13.399718 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6478c8b1-86af-4579-ba8e-97d1314b6ec8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:55:13.400332 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6478c8b1-86af-4579-ba8e-97d1314b6ec8 None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:55:13.400612 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6478c8b1-86af-4579-ba8e-97d1314b6ec8 None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:55:13.400889 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6478c8b1-86af-4579-ba8e-97d1314b6ec8 None admin] PATH_INFO: `/v3/users` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:55:13.401607 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6478c8b1-86af-4579-ba8e-97d1314b6ec8 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81438) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:55:13.401914 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6478c8b1-86af-4579-ba8e-97d1314b6ec8 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:55:13.403365 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6478c8b1-86af-4579-ba8e-97d1314b6ec8 None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:55:13.426004 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 526/1067] 10.222.0.50 () {56 vars in 1133 bytes} [Tue Sep 13 15:55:13 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:55:13.427660 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0dffefcc-520d-45c7-9388-f528f5d2ec9f None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:55:13.536159 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0dffefcc-520d-45c7-9388-f528f5d2ec9f None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:55:13.557278 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:55:13.568920 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2a7ad21-260f-4e13-a14a-316571232d1d None None] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:55:13.569706 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2a7ad21-260f-4e13-a14a-316571232d1d None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:55:13.569995 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2a7ad21-260f-4e13-a14a-316571232d1d None None] PATH_INFO: `/` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:55:13.571379 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 527/1068] 10.222.0.50 () {54 vars in 927 bytes} [Tue Sep 13 15:55:13 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 13 15:55:13.575987 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f90831ae-ff35-4ca1-a2ff-9b926470c943 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:55:13.576510 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0b54840-ffcd-4f92-8f3d-324c61ca294d None None] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:55:13.576619 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f90831ae-ff35-4ca1-a2ff-9b926470c943 None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:55:13.576789 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0b54840-ffcd-4f92-8f3d-324c61ca294d None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:55:13.576886 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f90831ae-ff35-4ca1-a2ff-9b926470c943 None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:55:13.577068 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0b54840-ffcd-4f92-8f3d-324c61ca294d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:55:13.577177 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f90831ae-ff35-4ca1-a2ff-9b926470c943 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:55:13.589679 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f90831ae-ff35-4ca1-a2ff-9b926470c943 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:55:13.591724 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f90831ae-ff35-4ca1-a2ff-9b926470c943 None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:55:13.596947 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.auth.core [None req-b0b54840-ffcd-4f92-8f3d-324c61ca294d None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81438) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:55:13.598141 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-b0b54840-ffcd-4f92-8f3d-324c61ca294d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:55:13.608907 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application [None req-f90831ae-ff35-4ca1-a2ff-9b926470c943 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:55:13.608907 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:55:13.608907 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:55:13.608907 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:55:13.608907 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:55:13.608907 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:55:13.608907 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:55:13.608907 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:55:13.608907 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:55:13.608907 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:55:13.608907 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:55:13.608907 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:55:13.608907 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 13 15:55:13.608907 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 13 15:55:13.608907 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 13 15:55:13.608907 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 13 15:55:13.609886 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:55:13.609886 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:55:13.609886 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:55:13.609886 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:55:13.609886 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:55:13.609886 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:55:13.609886 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:55:13.609886 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application with Lock( Sep 13 15:55:13.609886 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:55:13.609886 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:55:13.609886 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:55:13.609886 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:55:13.609886 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:55:13.609886 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:55:13.609886 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:55:13.609886 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:55:13.609886 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 13 15:55:13.610798 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 13 15:55:13.610798 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:55:13.610798 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application Sep 13 15:55:13.611080 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 542/1069] 10.222.0.50 () {56 vars in 1108 bytes} [Tue Sep 13 15:55:13 2022] GET /identity/v3/domains/Default => generated 87 bytes in 184 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:55:13.617012 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f90831ae-ff35-4ca1-a2ff-9b926470c943 None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:55:13.679813 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 528/1070] 10.222.0.50 () {58 vars in 1024 bytes} [Tue Sep 13 15:55:13 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 13 15:55:13.728285 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f90831ae-ff35-4ca1-a2ff-9b926470c943 None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:55:13.765939 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4446647e-cfcf-4745-b901-21efb57597fe None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:55:13.766565 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4446647e-cfcf-4745-b901-21efb57597fe None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:55:13.766844 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4446647e-cfcf-4745-b901-21efb57597fe None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:55:13.767149 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4446647e-cfcf-4745-b901-21efb57597fe None admin] PATH_INFO: `/v3/domains` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:55:13.767805 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4446647e-cfcf-4745-b901-21efb57597fe None admin] RBAC: Adding query filter params (name=Default) {{(pid=81437) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:55:13.768109 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4446647e-cfcf-4745-b901-21efb57597fe None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:55:13.769535 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4446647e-cfcf-4745-b901-21efb57597fe None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:55:13.791259 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 543/1071] 10.222.0.50 () {56 vars in 1109 bytes} [Tue Sep 13 15:55:13 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:55:13.793210 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0b54840-ffcd-4f92-8f3d-324c61ca294d None None] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:55:13.804586 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:55:13.864548 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e52d1343-f63f-4d90-891f-8ba15db5cf36 None None] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:55:13.864780 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e52d1343-f63f-4d90-891f-8ba15db5cf36 None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:55:13.865040 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e52d1343-f63f-4d90-891f-8ba15db5cf36 None None] PATH_INFO: `/` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:55:13.866916 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 544/1072] 10.222.0.50 () {54 vars in 927 bytes} [Tue Sep 13 15:55:13 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 13 15:55:13.871778 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-452a3773-1f73-4de1-b289-545e67f5eb22 None None] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:55:13.872056 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-452a3773-1f73-4de1-b289-545e67f5eb22 None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:55:13.872326 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-452a3773-1f73-4de1-b289-545e67f5eb22 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:55:13.891983 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.auth.core [None req-452a3773-1f73-4de1-b289-545e67f5eb22 None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81437) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:55:13.893151 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-452a3773-1f73-4de1-b289-545e67f5eb22 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:55:13.897145 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0b54840-ffcd-4f92-8f3d-324c61ca294d None None] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:55:13.921229 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:55:13.933123 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ac9e13f-0e4e-4e15-af83-0c34974d20a3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:55:13.934532 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ac9e13f-0e4e-4e15-af83-0c34974d20a3 None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:55:13.934813 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ac9e13f-0e4e-4e15-af83-0c34974d20a3 None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:55:13.935114 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ac9e13f-0e4e-4e15-af83-0c34974d20a3 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:55:13.942857 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ac9e13f-0e4e-4e15-af83-0c34974d20a3 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:55:13.944924 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ac9e13f-0e4e-4e15-af83-0c34974d20a3 None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:55:13.956198 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application [None req-6ac9e13f-0e4e-4e15-af83-0c34974d20a3 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 13 15:55:13.956198 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:55:13.956198 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:55:13.956198 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:55:13.956198 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:55:13.956198 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:55:13.956198 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:55:13.956198 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:55:13.956198 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:55:13.956198 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:55:13.956198 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:55:13.956198 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:55:13.956198 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 13 15:55:13.956198 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 13 15:55:13.956198 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 13 15:55:13.957219 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 13 15:55:13.957219 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:55:13.957219 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:55:13.957219 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:55:13.957219 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:55:13.957219 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:55:13.957219 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:55:13.957219 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:55:13.957219 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application with Lock( Sep 13 15:55:13.957219 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:55:13.957219 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:55:13.957219 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:55:13.957219 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:55:13.957219 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:55:13.957219 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:55:13.957219 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:55:13.957219 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:55:13.958573 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 13 15:55:13.958573 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 13 15:55:13.958573 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 13 15:55:13.958573 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 13 15:55:13.958573 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 13 15:55:13.958573 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 13 15:55:13.958573 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 13 15:55:13.958573 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Sep 13 15:55:13.958573 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 529/1073] 10.222.0.50 () {56 vars in 1111 bytes} [Tue Sep 13 15:55:13 2022] GET /identity/v3/projects/service => generated 88 bytes in 165 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:55:13.958573 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ac9e13f-0e4e-4e15-af83-0c34974d20a3 None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:55:13.969941 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 545/1074] 10.222.0.50 () {58 vars in 1024 bytes} [Tue Sep 13 15:55:13 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 13 15:55:13.970848 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-452a3773-1f73-4de1-b289-545e67f5eb22 None None] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:55:14.033532 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:55:14.076958 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ac9e13f-0e4e-4e15-af83-0c34974d20a3 None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:55:14.084457 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-452a3773-1f73-4de1-b289-545e67f5eb22 None None] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:55:14.110604 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9359e2cf-f777-4962-b012-3f0bfc2c71a9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:55:14.111264 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9359e2cf-f777-4962-b012-3f0bfc2c71a9 None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:55:14.111542 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9359e2cf-f777-4962-b012-3f0bfc2c71a9 None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:55:14.111814 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9359e2cf-f777-4962-b012-3f0bfc2c71a9 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:55:14.122360 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9359e2cf-f777-4962-b012-3f0bfc2c71a9 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 13 15:55:14.123696 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ce206b9-72ed-4c4b-801e-b53510131951 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:55:14.124876 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ce206b9-72ed-4c4b-801e-b53510131951 None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:55:14.125336 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ce206b9-72ed-4c4b-801e-b53510131951 None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:55:14.125786 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ce206b9-72ed-4c4b-801e-b53510131951 None admin] PATH_INFO: `/v3/projects` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:55:14.126657 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ce206b9-72ed-4c4b-801e-b53510131951 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81437) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:55:14.127169 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ce206b9-72ed-4c4b-801e-b53510131951 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:55:14.127548 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application [None req-9359e2cf-f777-4962-b012-3f0bfc2c71a9 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 13 15:55:14.127548 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:55:14.127548 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:55:14.127548 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:55:14.127548 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:55:14.127548 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:55:14.127548 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:55:14.127548 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:55:14.127548 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:55:14.127548 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:55:14.127548 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:55:14.127548 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:55:14.127548 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 13 15:55:14.127548 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 13 15:55:14.127548 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 13 15:55:14.127548 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 13 15:55:14.128503 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 13 15:55:14.128503 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 13 15:55:14.128503 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 13 15:55:14.128503 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 13 15:55:14.128503 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:55:14.128503 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:55:14.128503 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:55:14.128503 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:55:14.128503 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:55:14.128503 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:55:14.128503 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:55:14.128503 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application with Lock( Sep 13 15:55:14.128503 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:55:14.128503 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:55:14.128503 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:55:14.128503 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:55:14.129370 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:55:14.129370 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:55:14.129370 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:55:14.129370 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:55:14.129370 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 13 15:55:14.129370 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 13 15:55:14.129370 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 13 15:55:14.129370 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 13 15:55:14.129370 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 13 15:55:14.129370 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 13 15:55:14.129370 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 13 15:55:14.129370 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Sep 13 15:55:14.129370 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 530/1075] 10.222.0.50 () {56 vars in 1096 bytes} [Tue Sep 13 15:55:13 2022] GET /identity/v3/roles/admin => generated 83 bytes in 171 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:55:14.130556 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ce206b9-72ed-4c4b-801e-b53510131951 None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:55:14.134540 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9359e2cf-f777-4962-b012-3f0bfc2c71a9 None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:55:14.148847 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 546/1076] 10.222.0.50 () {56 vars in 1148 bytes} [Tue Sep 13 15:55:13 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:55:14.149860 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ce206b9-72ed-4c4b-801e-b53510131951 None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:55:14.175640 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:55:14.239018 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9359e2cf-f777-4962-b012-3f0bfc2c71a9 None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:55:14.260361 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ce206b9-72ed-4c4b-801e-b53510131951 None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:55:14.266504 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:55:14.275362 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57dcfe64-25be-422e-afbd-083eb36dc71e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:55:14.275968 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57dcfe64-25be-422e-afbd-083eb36dc71e None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:55:14.276253 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57dcfe64-25be-422e-afbd-083eb36dc71e None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:55:14.276523 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57dcfe64-25be-422e-afbd-083eb36dc71e None admin] PATH_INFO: `/v3/roles` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:55:14.277175 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57dcfe64-25be-422e-afbd-083eb36dc71e None admin] RBAC: Adding query filter params (name=admin) {{(pid=81438) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:55:14.277474 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57dcfe64-25be-422e-afbd-083eb36dc71e None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:55:14.278821 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57dcfe64-25be-422e-afbd-083eb36dc71e None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:55:14.292494 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 531/1077] 10.222.0.50 () {56 vars in 1099 bytes} [Tue Sep 13 15:55:14 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 13 15:55:14.292494 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57dcfe64-25be-422e-afbd-083eb36dc71e None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:55:14.294155 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55ddcc85-bf07-48c3-ae59-221d1986dcc8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:55:14.294810 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55ddcc85-bf07-48c3-ae59-221d1986dcc8 None admin] REQUEST_METHOD: `PUT` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:55:14.295115 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55ddcc85-bf07-48c3-ae59-221d1986dcc8 None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:55:14.295387 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55ddcc85-bf07-48c3-ae59-221d1986dcc8 None admin] PATH_INFO: `/v3/projects/5811e3b83a1e462e9320cad79a3ad791/users/4e039b46c5e3445bbb39904b00c5430e/roles/20e2f11234344900a51fc76fe5b625f3` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:55:14.297501 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55ddcc85-bf07-48c3-ae59-221d1986dcc8 None admin] RBAC: Authorizing `identity:create_grant(project_id=5811e3b83a1e462e9320cad79a3ad791, user_id=4e039b46c5e3445bbb39904b00c5430e, role_id=20e2f11234344900a51fc76fe5b625f3)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:55:14.298933 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55ddcc85-bf07-48c3-ae59-221d1986dcc8 None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:55:14.312713 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 547/1078] 10.222.0.50 () {58 vars in 1439 bytes} [Tue Sep 13 15:55:14 2022] PUT /identity/v3/projects/5811e3b83a1e462e9320cad79a3ad791/users/4e039b46c5e3445bbb39904b00c5430e/roles/20e2f11234344900a51fc76fe5b625f3 => generated 0 bytes in 163 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 13 15:55:14.323194 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55ddcc85-bf07-48c3-ae59-221d1986dcc8 None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:55:14.401811 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57dcfe64-25be-422e-afbd-083eb36dc71e None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:55:14.418343 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:55:14.438544 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55ddcc85-bf07-48c3-ae59-221d1986dcc8 None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:55:14.445437 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebbb809d-b4d8-48f2-b538-3db7a39b4ab3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:55:14.446359 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebbb809d-b4d8-48f2-b538-3db7a39b4ab3 None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:55:14.446964 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebbb809d-b4d8-48f2-b538-3db7a39b4ab3 None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:55:14.447246 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebbb809d-b4d8-48f2-b538-3db7a39b4ab3 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:55:14.475104 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-220d704e-bc61-46a8-a77f-e722b50b855b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:55:14.475767 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-220d704e-bc61-46a8-a77f-e722b50b855b None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:55:14.476042 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-220d704e-bc61-46a8-a77f-e722b50b855b None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:55:14.476389 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-220d704e-bc61-46a8-a77f-e722b50b855b None admin] PATH_INFO: `/v3/users/4d24fa39b5ff4de7a57641b0bf2c1044` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:55:14.477509 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-220d704e-bc61-46a8-a77f-e722b50b855b None admin] RBAC: Authorizing `identity:get_user(user_id=4d24fa39b5ff4de7a57641b0bf2c1044)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:55:14.479003 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-220d704e-bc61-46a8-a77f-e722b50b855b None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:55:14.485507 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 548/1079] 10.222.0.50 () {56 vars in 1177 bytes} [Tue Sep 13 15:55:14 2022] GET /identity/v3/users/4d24fa39b5ff4de7a57641b0bf2c1044 => generated 285 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:55:14.488268 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-220d704e-bc61-46a8-a77f-e722b50b855b None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:55:14.517323 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:55:14.531055 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebbb809d-b4d8-48f2-b538-3db7a39b4ab3 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:55:14.532830 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebbb809d-b4d8-48f2-b538-3db7a39b4ab3 None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:55:14.590948 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-220d704e-bc61-46a8-a77f-e722b50b855b None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:55:14.627513 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1682636-c93d-419f-996d-87177ffc2528 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:55:14.628144 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1682636-c93d-419f-996d-87177ffc2528 None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:55:14.628424 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1682636-c93d-419f-996d-87177ffc2528 None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:55:14.628696 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1682636-c93d-419f-996d-87177ffc2528 None admin] PATH_INFO: `/v3/projects/13d63282b1fd48349397eed6fccbd62a` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:55:14.629819 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1682636-c93d-419f-996d-87177ffc2528 None admin] RBAC: Authorizing `identity:get_project(project_id=13d63282b1fd48349397eed6fccbd62a)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:55:14.631291 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1682636-c93d-419f-996d-87177ffc2528 None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:55:14.633224 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 549/1080] 10.222.0.50 () {56 vars in 1186 bytes} [Tue Sep 13 15:55:14 2022] GET /identity/v3/projects/13d63282b1fd48349397eed6fccbd62a => generated 311 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:55:14.638749 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1682636-c93d-419f-996d-87177ffc2528 None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:55:14.647325 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 532/1081] 10.222.0.50 () {58 vars in 1282 bytes} [Tue Sep 13 15:55:14 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 359 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 13 15:55:14.652866 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebbb809d-b4d8-48f2-b538-3db7a39b4ab3 None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:55:14.659685 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:55:14.748168 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1682636-c93d-419f-996d-87177ffc2528 None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:55:14.756234 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:55:14.767591 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebbb809d-b4d8-48f2-b538-3db7a39b4ab3 None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:55:14.782715 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94302856-b662-480b-aefe-42dcef890da1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:55:14.783406 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94302856-b662-480b-aefe-42dcef890da1 None admin] REQUEST_METHOD: `PUT` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:55:14.783722 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94302856-b662-480b-aefe-42dcef890da1 None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:55:14.783957 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94302856-b662-480b-aefe-42dcef890da1 None admin] PATH_INFO: `/v3/projects/13d63282b1fd48349397eed6fccbd62a/users/4d24fa39b5ff4de7a57641b0bf2c1044/roles/031d4e1a23e34f71858fc4a6dfbd4ccb` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:55:14.785862 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94302856-b662-480b-aefe-42dcef890da1 None admin] RBAC: Authorizing `identity:create_grant(project_id=13d63282b1fd48349397eed6fccbd62a, user_id=4d24fa39b5ff4de7a57641b0bf2c1044, role_id=031d4e1a23e34f71858fc4a6dfbd4ccb)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:55:14.788007 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94302856-b662-480b-aefe-42dcef890da1 None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:55:14.802952 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 550/1082] 10.222.0.50 () {58 vars in 1439 bytes} [Tue Sep 13 15:55:14 2022] PUT /identity/v3/projects/13d63282b1fd48349397eed6fccbd62a/users/4d24fa39b5ff4de7a57641b0bf2c1044/roles/031d4e1a23e34f71858fc4a6dfbd4ccb => generated 0 bytes in 164 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:55:14.811511 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5cd9ec6-467b-4454-bd0c-f33e7d180ebb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:55:14.812275 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5cd9ec6-467b-4454-bd0c-f33e7d180ebb None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:55:14.812674 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5cd9ec6-467b-4454-bd0c-f33e7d180ebb None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:55:14.813023 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5cd9ec6-467b-4454-bd0c-f33e7d180ebb None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:55:14.826552 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5cd9ec6-467b-4454-bd0c-f33e7d180ebb None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:55:14.828178 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5cd9ec6-467b-4454-bd0c-f33e7d180ebb None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:55:14.847517 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application [None req-d5cd9ec6-467b-4454-bd0c-f33e7d180ebb None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:55:14.847517 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:55:14.847517 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:55:14.847517 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:55:14.847517 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:55:14.847517 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:55:14.847517 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:55:14.847517 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:55:14.847517 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:55:14.847517 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:55:14.847517 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:55:14.847517 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:55:14.847517 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 13 15:55:14.847517 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 13 15:55:14.847517 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 13 15:55:14.847517 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 13 15:55:14.848572 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:55:14.848572 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:55:14.848572 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:55:14.848572 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:55:14.848572 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:55:14.848572 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:55:14.848572 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:55:14.848572 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application with Lock( Sep 13 15:55:14.848572 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:55:14.848572 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:55:14.848572 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:55:14.848572 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:55:14.848572 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:55:14.848572 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:55:14.848572 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:55:14.848572 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:55:14.848572 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 13 15:55:14.849622 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 13 15:55:14.849622 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:55:14.849622 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Sep 13 15:55:14.849622 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 533/1083] 10.222.0.50 () {56 vars in 1108 bytes} [Tue Sep 13 15:55:14 2022] GET /identity/v3/domains/Default => generated 87 bytes in 196 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 13 15:55:14.856230 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94302856-b662-480b-aefe-42dcef890da1 None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:55:14.953769 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:55:14.982443 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94302856-b662-480b-aefe-42dcef890da1 None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:55:15.017943 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a82c241-4b8f-4fa4-837a-77db18babadb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:55:15.018554 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a82c241-4b8f-4fa4-837a-77db18babadb None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:55:15.018831 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a82c241-4b8f-4fa4-837a-77db18babadb None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:55:15.019141 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a82c241-4b8f-4fa4-837a-77db18babadb None admin] PATH_INFO: `/v3/domains` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:55:15.019785 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a82c241-4b8f-4fa4-837a-77db18babadb None admin] RBAC: Adding query filter params (name=Default) {{(pid=81437) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:55:15.020087 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a82c241-4b8f-4fa4-837a-77db18babadb None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:55:15.021452 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a82c241-4b8f-4fa4-837a-77db18babadb None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:55:15.034417 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 551/1084] 10.222.0.50 () {56 vars in 1109 bytes} [Tue Sep 13 15:55:14 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:55:16.501969 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85b01173-8f0a-4fce-b480-9f0d346c2418 None None] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:55:16.502494 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85b01173-8f0a-4fce-b480-9f0d346c2418 None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:55:16.502494 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85b01173-8f0a-4fce-b480-9f0d346c2418 None None] PATH_INFO: `/` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:55:16.503873 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 534/1085] 10.222.0.50 () {54 vars in 927 bytes} [Tue Sep 13 15:55:16 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:55:16.509637 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ef8c778-6724-4caf-aa93-2b75320f3f61 None None] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:55:16.510098 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ef8c778-6724-4caf-aa93-2b75320f3f61 None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:55:16.510198 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ef8c778-6724-4caf-aa93-2b75320f3f61 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:55:16.529035 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.auth.core [None req-2ef8c778-6724-4caf-aa93-2b75320f3f61 None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81437) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:55:16.530222 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-2ef8c778-6724-4caf-aa93-2b75320f3f61 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:55:16.596415 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 552/1086] 10.222.0.50 () {58 vars in 1024 bytes} [Tue Sep 13 15:55:16 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 13 15:55:16.873435 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85b01173-8f0a-4fce-b480-9f0d346c2418 None None] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:55:16.960862 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3d6b137-2900-4675-a37a-e8c16583bef2 None None] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:55:16.961147 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3d6b137-2900-4675-a37a-e8c16583bef2 None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:55:16.961426 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3d6b137-2900-4675-a37a-e8c16583bef2 None None] PATH_INFO: `/` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:55:16.962768 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 553/1087] 10.222.0.50 () {54 vars in 927 bytes} [Tue Sep 13 15:55:16 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 13 15:55:16.967785 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0155e9e-195a-42e4-bc34-dd82b96c3f56 None None] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:55:16.968069 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0155e9e-195a-42e4-bc34-dd82b96c3f56 None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:55:16.968344 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0155e9e-195a-42e4-bc34-dd82b96c3f56 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:55:16.975995 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85b01173-8f0a-4fce-b480-9f0d346c2418 None None] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:55:16.985997 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.auth.core [None req-d0155e9e-195a-42e4-bc34-dd82b96c3f56 None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81437) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:55:16.987528 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-d0155e9e-195a-42e4-bc34-dd82b96c3f56 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:55:17.010563 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9d85efe-f752-4090-888c-7da67cdf2de4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:55:17.011221 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9d85efe-f752-4090-888c-7da67cdf2de4 None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:55:17.011503 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9d85efe-f752-4090-888c-7da67cdf2de4 None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:55:17.011778 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9d85efe-f752-4090-888c-7da67cdf2de4 None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:55:17.022677 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9d85efe-f752-4090-888c-7da67cdf2de4 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 13 15:55:17.027443 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application [None req-a9d85efe-f752-4090-888c-7da67cdf2de4 None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Sep 13 15:55:17.027443 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:55:17.027443 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:55:17.027443 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:55:17.027443 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:55:17.027443 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:55:17.027443 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:55:17.027443 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:55:17.027443 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:55:17.027443 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:55:17.027443 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:55:17.027443 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:55:17.027443 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 13 15:55:17.027443 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 13 15:55:17.027443 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 13 15:55:17.027443 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 13 15:55:17.031025 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 13 15:55:17.031025 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 13 15:55:17.031025 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 13 15:55:17.031025 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 13 15:55:17.031025 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:55:17.031025 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:55:17.031025 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:55:17.031025 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:55:17.031025 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:55:17.031025 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:55:17.031025 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:55:17.031025 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application with Lock( Sep 13 15:55:17.031025 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:55:17.031025 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:55:17.031025 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:55:17.031025 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:55:17.032007 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:55:17.032007 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:55:17.032007 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:55:17.032007 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:55:17.032007 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 13 15:55:17.032007 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 13 15:55:17.032007 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 13 15:55:17.032007 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 13 15:55:17.032007 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 13 15:55:17.032007 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 13 15:55:17.032007 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Sep 13 15:55:17.032007 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Sep 13 15:55:17.032007 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 535/1088] 10.222.0.50 () {56 vars in 1120 bytes} [Tue Sep 13 15:55:16 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 156 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:55:17.034462 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9d85efe-f752-4090-888c-7da67cdf2de4 None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:55:17.037088 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:55:17.049577 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:55:17.056259 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 554/1089] 10.222.0.50 () {58 vars in 1024 bytes} [Tue Sep 13 15:55:16 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 13 15:55:17.135335 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9d85efe-f752-4090-888c-7da67cdf2de4 None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:55:17.167480 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee2bcf57-46fe-4e5d-816d-8e7a3961f8df None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:55:17.168082 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee2bcf57-46fe-4e5d-816d-8e7a3961f8df None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:55:17.168358 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee2bcf57-46fe-4e5d-816d-8e7a3961f8df None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:55:17.168632 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee2bcf57-46fe-4e5d-816d-8e7a3961f8df None admin] PATH_INFO: `/v3/roles` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:55:17.169280 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee2bcf57-46fe-4e5d-816d-8e7a3961f8df None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=81438) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:55:17.169583 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee2bcf57-46fe-4e5d-816d-8e7a3961f8df None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:55:17.170963 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee2bcf57-46fe-4e5d-816d-8e7a3961f8df None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:55:17.179308 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 536/1090] 10.222.0.50 () {56 vars in 1115 bytes} [Tue Sep 13 15:55:17 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 145 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 13 15:55:17.185367 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0155e9e-195a-42e4-bc34-dd82b96c3f56 None None] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:55:17.282198 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0155e9e-195a-42e4-bc34-dd82b96c3f56 None None] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:55:17.315346 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06387ab9-6dff-4af6-b4ab-18cbea85c3e4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:55:17.315969 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06387ab9-6dff-4af6-b4ab-18cbea85c3e4 None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:55:17.316247 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06387ab9-6dff-4af6-b4ab-18cbea85c3e4 None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:55:17.316527 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06387ab9-6dff-4af6-b4ab-18cbea85c3e4 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:55:17.322276 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee2bcf57-46fe-4e5d-816d-8e7a3961f8df None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:55:17.327099 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06387ab9-6dff-4af6-b4ab-18cbea85c3e4 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:55:17.328601 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06387ab9-6dff-4af6-b4ab-18cbea85c3e4 None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:55:17.344076 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application [None req-06387ab9-6dff-4af6-b4ab-18cbea85c3e4 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:55:17.344076 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:55:17.344076 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:55:17.344076 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:55:17.344076 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:55:17.344076 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:55:17.344076 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:55:17.344076 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:55:17.344076 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:55:17.344076 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:55:17.344076 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:55:17.344076 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:55:17.344076 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 13 15:55:17.344076 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 13 15:55:17.344076 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 13 15:55:17.344076 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 13 15:55:17.345214 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:55:17.345214 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:55:17.345214 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:55:17.345214 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:55:17.345214 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:55:17.345214 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:55:17.345214 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:55:17.345214 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application with Lock( Sep 13 15:55:17.345214 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:55:17.345214 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:55:17.345214 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:55:17.345214 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:55:17.345214 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:55:17.345214 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:55:17.345214 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:55:17.345214 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:55:17.345214 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 13 15:55:17.346634 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 13 15:55:17.346634 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:55:17.346634 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application Sep 13 15:55:17.346634 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 555/1091] 10.222.0.50 () {56 vars in 1108 bytes} [Tue Sep 13 15:55:17 2022] GET /identity/v3/domains/Default => generated 87 bytes in 160 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 13 15:55:17.351101 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06387ab9-6dff-4af6-b4ab-18cbea85c3e4 None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:55:17.387828 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:55:17.402066 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:55:17.434470 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee2bcf57-46fe-4e5d-816d-8e7a3961f8df None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:55:17.452622 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06387ab9-6dff-4af6-b4ab-18cbea85c3e4 None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:55:17.468885 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82fbb856-c472-4bdf-bcc9-d122aff28c42 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:55:17.469515 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82fbb856-c472-4bdf-bcc9-d122aff28c42 None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:55:17.469794 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82fbb856-c472-4bdf-bcc9-d122aff28c42 None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:55:17.470068 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82fbb856-c472-4bdf-bcc9-d122aff28c42 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:55:17.480749 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82fbb856-c472-4bdf-bcc9-d122aff28c42 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 13 15:55:17.485430 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-761f7aa8-7ec6-4906-a1dd-e6ec3b380671 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:55:17.485582 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application [None req-82fbb856-c472-4bdf-bcc9-d122aff28c42 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 13 15:55:17.485582 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:55:17.485582 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:55:17.485582 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:55:17.485582 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:55:17.485582 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:55:17.485582 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:55:17.485582 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:55:17.485582 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:55:17.485582 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:55:17.485582 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:55:17.485582 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:55:17.485582 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 13 15:55:17.485582 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 13 15:55:17.485582 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 13 15:55:17.485582 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 13 15:55:17.486580 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 13 15:55:17.486580 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 13 15:55:17.486580 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 13 15:55:17.486580 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 13 15:55:17.486580 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:55:17.486580 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:55:17.486580 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:55:17.486580 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:55:17.486580 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:55:17.486580 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:55:17.486580 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:55:17.486580 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application with Lock( Sep 13 15:55:17.486580 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:55:17.486580 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:55:17.486580 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:55:17.486580 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:55:17.487516 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:55:17.487516 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:55:17.487516 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:55:17.487516 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:55:17.487516 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 13 15:55:17.487516 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 13 15:55:17.487516 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 13 15:55:17.487516 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 13 15:55:17.487516 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 13 15:55:17.487516 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 13 15:55:17.487516 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 13 15:55:17.487516 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Sep 13 15:55:17.488178 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-761f7aa8-7ec6-4906-a1dd-e6ec3b380671 None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:55:17.488178 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-761f7aa8-7ec6-4906-a1dd-e6ec3b380671 None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:55:17.488178 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-761f7aa8-7ec6-4906-a1dd-e6ec3b380671 None admin] PATH_INFO: `/v3/domains` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:55:17.488404 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 537/1092] 10.222.0.50 () {56 vars in 1096 bytes} [Tue Sep 13 15:55:17 2022] GET /identity/v3/roles/admin => generated 83 bytes in 165 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 13 15:55:17.488488 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-761f7aa8-7ec6-4906-a1dd-e6ec3b380671 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81437) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:55:17.488488 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-761f7aa8-7ec6-4906-a1dd-e6ec3b380671 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:55:17.488903 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-761f7aa8-7ec6-4906-a1dd-e6ec3b380671 None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:55:17.493046 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82fbb856-c472-4bdf-bcc9-d122aff28c42 None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:55:17.501146 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 556/1093] 10.222.0.50 () {56 vars in 1109 bytes} [Tue Sep 13 15:55:17 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:55:17.507657 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-761f7aa8-7ec6-4906-a1dd-e6ec3b380671 None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:55:17.592368 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82fbb856-c472-4bdf-bcc9-d122aff28c42 None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:55:17.605940 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-761f7aa8-7ec6-4906-a1dd-e6ec3b380671 None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:55:17.616231 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:55:17.616652 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:55:17.627334 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9dcb6aef-eac5-4b43-9051-993f90d041a5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:55:17.627938 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dcb6aef-eac5-4b43-9051-993f90d041a5 None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:55:17.628215 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dcb6aef-eac5-4b43-9051-993f90d041a5 None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:55:17.628485 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dcb6aef-eac5-4b43-9051-993f90d041a5 None admin] PATH_INFO: `/v3/roles` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:55:17.629137 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9dcb6aef-eac5-4b43-9051-993f90d041a5 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81438) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:55:17.629434 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9dcb6aef-eac5-4b43-9051-993f90d041a5 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:55:17.630810 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9dcb6aef-eac5-4b43-9051-993f90d041a5 None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:55:17.639405 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 538/1094] 10.222.0.50 () {56 vars in 1099 bytes} [Tue Sep 13 15:55:17 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 13 15:55:17.640930 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbe28aff-aa54-4841-9f8c-c14b8e9a8fec None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:55:17.641653 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbe28aff-aa54-4841-9f8c-c14b8e9a8fec None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:55:17.642003 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbe28aff-aa54-4841-9f8c-c14b8e9a8fec None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:55:17.642352 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbe28aff-aa54-4841-9f8c-c14b8e9a8fec None admin] PATH_INFO: `/v3/users/nova` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:55:17.645096 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9dcb6aef-eac5-4b43-9051-993f90d041a5 None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:55:17.650610 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbe28aff-aa54-4841-9f8c-c14b8e9a8fec None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:55:17.652105 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbe28aff-aa54-4841-9f8c-c14b8e9a8fec None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:55:17.664193 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application [None req-dbe28aff-aa54-4841-9f8c-c14b8e9a8fec None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Sep 13 15:55:17.664193 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:55:17.664193 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:55:17.664193 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:55:17.664193 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:55:17.664193 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:55:17.664193 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:55:17.664193 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:55:17.664193 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:55:17.664193 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:55:17.664193 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:55:17.664193 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:55:17.664193 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 13 15:55:17.664193 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 13 15:55:17.664193 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 13 15:55:17.664193 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 13 15:55:17.665151 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:55:17.665151 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:55:17.665151 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 13 15:55:17.665151 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 13 15:55:17.665151 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 13 15:55:17.665151 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 13 15:55:17.665151 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:55:17.665151 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:55:17.665151 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:55:17.665151 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:55:17.665151 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:55:17.665151 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application with Lock( Sep 13 15:55:17.665151 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:55:17.665151 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:55:17.665151 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:55:17.665151 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:55:17.665151 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:55:17.665151 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:55:17.666132 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:55:17.666132 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:55:17.666132 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 13 15:55:17.666132 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 13 15:55:17.666132 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 13 15:55:17.666132 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 13 15:55:17.666132 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 13 15:55:17.666132 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 13 15:55:17.666132 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Sep 13 15:55:17.666132 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application Sep 13 15:55:17.666132 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 557/1095] 10.222.0.50 () {56 vars in 1093 bytes} [Tue Sep 13 15:55:17 2022] GET /identity/v3/users/nova => generated 82 bytes in 158 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 13 15:55:17.670916 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbe28aff-aa54-4841-9f8c-c14b8e9a8fec None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:55:17.747402 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9dcb6aef-eac5-4b43-9051-993f90d041a5 None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:55:17.771409 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbe28aff-aa54-4841-9f8c-c14b8e9a8fec None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:55:17.780426 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb4ab82c-6b97-4d06-9930-57dad64f544d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:55:17.781050 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb4ab82c-6b97-4d06-9930-57dad64f544d None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:55:17.781332 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb4ab82c-6b97-4d06-9930-57dad64f544d None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:55:17.781599 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb4ab82c-6b97-4d06-9930-57dad64f544d None admin] PATH_INFO: `/v3/users/4d24fa39b5ff4de7a57641b0bf2c1044` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:55:17.782717 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb4ab82c-6b97-4d06-9930-57dad64f544d None admin] RBAC: Authorizing `identity:get_user(user_id=4d24fa39b5ff4de7a57641b0bf2c1044)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:55:17.784174 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb4ab82c-6b97-4d06-9930-57dad64f544d None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:55:17.785904 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 539/1096] 10.222.0.50 () {56 vars in 1177 bytes} [Tue Sep 13 15:55:17 2022] GET /identity/v3/users/4d24fa39b5ff4de7a57641b0bf2c1044 => generated 285 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 13 15:55:17.791617 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb4ab82c-6b97-4d06-9930-57dad64f544d None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:55:17.805536 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65251533-c9ce-4530-8630-166eec2859d4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:55:17.806274 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65251533-c9ce-4530-8630-166eec2859d4 None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:55:17.806560 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65251533-c9ce-4530-8630-166eec2859d4 None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:55:17.807135 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65251533-c9ce-4530-8630-166eec2859d4 None admin] PATH_INFO: `/v3/users` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:55:17.807944 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65251533-c9ce-4530-8630-166eec2859d4 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81437) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:55:17.808253 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65251533-c9ce-4530-8630-166eec2859d4 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:55:17.810610 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65251533-c9ce-4530-8630-166eec2859d4 None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:55:17.832039 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 558/1097] 10.222.0.50 () {56 vars in 1133 bytes} [Tue Sep 13 15:55:17 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 13 15:55:17.837952 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65251533-c9ce-4530-8630-166eec2859d4 None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:55:17.852098 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:55:17.866503 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:55:17.896866 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb4ab82c-6b97-4d06-9930-57dad64f544d None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:55:17.930676 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74b1f5bc-3495-418f-8653-71f2b0fb1674 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:55:17.931332 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74b1f5bc-3495-418f-8653-71f2b0fb1674 None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:55:17.931879 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74b1f5bc-3495-418f-8653-71f2b0fb1674 None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:55:17.932159 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74b1f5bc-3495-418f-8653-71f2b0fb1674 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:55:17.947418 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65251533-c9ce-4530-8630-166eec2859d4 None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:55:17.990980 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36aaefd7-81e4-4cd0-8b8a-991b1ded3ea6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:55:17.991965 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36aaefd7-81e4-4cd0-8b8a-991b1ded3ea6 None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:55:17.992452 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36aaefd7-81e4-4cd0-8b8a-991b1ded3ea6 None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:55:17.992907 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36aaefd7-81e4-4cd0-8b8a-991b1ded3ea6 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:55:18.006781 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74b1f5bc-3495-418f-8653-71f2b0fb1674 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:55:18.008362 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74b1f5bc-3495-418f-8653-71f2b0fb1674 None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:55:18.066001 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:55:18.074263 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36aaefd7-81e4-4cd0-8b8a-991b1ded3ea6 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:55:18.075749 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36aaefd7-81e4-4cd0-8b8a-991b1ded3ea6 None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:55:18.105268 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:55:18.131143 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 540/1098] 10.222.0.50 () {58 vars in 1282 bytes} [Tue Sep 13 15:55:17 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 340 msecs (HTTP/1.1 200) 6 headers in 359 bytes (2 switches on core 0) Sep 13 15:55:18.137304 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74b1f5bc-3495-418f-8653-71f2b0fb1674 None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:55:18.190000 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 559/1099] 10.222.0.50 () {58 vars in 1282 bytes} [Tue Sep 13 15:55:17 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 351 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 13 15:55:18.194964 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36aaefd7-81e4-4cd0-8b8a-991b1ded3ea6 None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:55:18.263624 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74b1f5bc-3495-418f-8653-71f2b0fb1674 None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:55:18.307938 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d84c7be2-c0cd-4500-9a66-bc9db7f116a8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:55:18.308576 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d84c7be2-c0cd-4500-9a66-bc9db7f116a8 None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:55:18.308855 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d84c7be2-c0cd-4500-9a66-bc9db7f116a8 None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:55:18.309132 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d84c7be2-c0cd-4500-9a66-bc9db7f116a8 None admin] PATH_INFO: `/v3/projects/13d63282b1fd48349397eed6fccbd62a` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:55:18.310786 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d84c7be2-c0cd-4500-9a66-bc9db7f116a8 None admin] RBAC: Authorizing `identity:get_project(project_id=13d63282b1fd48349397eed6fccbd62a)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:55:18.312254 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d84c7be2-c0cd-4500-9a66-bc9db7f116a8 None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:55:18.314258 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 541/1100] 10.222.0.50 () {56 vars in 1186 bytes} [Tue Sep 13 15:55:18 2022] GET /identity/v3/projects/13d63282b1fd48349397eed6fccbd62a => generated 311 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 13 15:55:18.320342 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d84c7be2-c0cd-4500-9a66-bc9db7f116a8 None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:55:18.322820 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36aaefd7-81e4-4cd0-8b8a-991b1ded3ea6 None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:55:18.347447 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:55:18.354776 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:55:18.363408 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3062a0c9-693b-408e-a843-28ec95b4b40d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:55:18.364107 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3062a0c9-693b-408e-a843-28ec95b4b40d None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:55:18.364391 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3062a0c9-693b-408e-a843-28ec95b4b40d None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:55:18.364666 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3062a0c9-693b-408e-a843-28ec95b4b40d None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:55:18.374997 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3062a0c9-693b-408e-a843-28ec95b4b40d None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:55:18.376457 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3062a0c9-693b-408e-a843-28ec95b4b40d None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:55:18.391258 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application [None req-3062a0c9-693b-408e-a843-28ec95b4b40d None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:55:18.391258 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:55:18.391258 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:55:18.391258 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:55:18.391258 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:55:18.391258 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:55:18.391258 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:55:18.391258 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:55:18.391258 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:55:18.391258 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:55:18.391258 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:55:18.391258 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:55:18.391258 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 13 15:55:18.391258 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 13 15:55:18.391258 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 13 15:55:18.391258 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 13 15:55:18.392288 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:55:18.392288 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:55:18.392288 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:55:18.392288 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:55:18.392288 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:55:18.392288 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:55:18.392288 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:55:18.392288 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application with Lock( Sep 13 15:55:18.392288 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:55:18.392288 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:55:18.392288 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:55:18.392288 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:55:18.392288 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:55:18.392288 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:55:18.392288 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:55:18.392288 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:55:18.392288 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 13 15:55:18.393216 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 13 15:55:18.393216 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:55:18.393216 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application Sep 13 15:55:18.393216 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 560/1101] 10.222.0.50 () {56 vars in 1108 bytes} [Tue Sep 13 15:55:18 2022] GET /identity/v3/domains/Default => generated 87 bytes in 198 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:55:18.398945 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3062a0c9-693b-408e-a843-28ec95b4b40d None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:55:18.441882 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d84c7be2-c0cd-4500-9a66-bc9db7f116a8 None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:55:18.486063 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3179bdb7-4679-4d55-a0f3-0b90b41414c3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:55:18.486671 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3179bdb7-4679-4d55-a0f3-0b90b41414c3 None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:55:18.486978 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3179bdb7-4679-4d55-a0f3-0b90b41414c3 None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:55:18.487252 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3179bdb7-4679-4d55-a0f3-0b90b41414c3 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:55:18.487970 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3179bdb7-4679-4d55-a0f3-0b90b41414c3 None admin] RBAC: Adding query filter params (role.id=031d4e1a23e34f71858fc4a6dfbd4ccb, scope.project.id=13d63282b1fd48349397eed6fccbd62a, user.id=4d24fa39b5ff4de7a57641b0bf2c1044) {{(pid=81438) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:55:18.488278 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3179bdb7-4679-4d55-a0f3-0b90b41414c3 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:55:18.490055 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3179bdb7-4679-4d55-a0f3-0b90b41414c3 None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:55:18.510933 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 542/1102] 10.222.0.50 () {56 vars in 1374 bytes} [Tue Sep 13 15:55:18 2022] GET /identity/v3/role_assignments?user.id=4d24fa39b5ff4de7a57641b0bf2c1044&scope.project.id=13d63282b1fd48349397eed6fccbd62a&role.id=031d4e1a23e34f71858fc4a6dfbd4ccb => generated 608 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 13 15:55:18.524516 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3062a0c9-693b-408e-a843-28ec95b4b40d None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:55:18.572558 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-553ce2cd-8ebb-4c9a-ae61-3a25336d0097 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:55:18.573214 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-553ce2cd-8ebb-4c9a-ae61-3a25336d0097 None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:55:18.573647 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-553ce2cd-8ebb-4c9a-ae61-3a25336d0097 None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:55:18.574153 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-553ce2cd-8ebb-4c9a-ae61-3a25336d0097 None admin] PATH_INFO: `/v3/domains` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:55:18.575133 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-553ce2cd-8ebb-4c9a-ae61-3a25336d0097 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81437) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:55:18.575447 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-553ce2cd-8ebb-4c9a-ae61-3a25336d0097 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:55:18.577446 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-553ce2cd-8ebb-4c9a-ae61-3a25336d0097 None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:55:18.596494 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 561/1103] 10.222.0.50 () {56 vars in 1109 bytes} [Tue Sep 13 15:55:18 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 193 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 13 15:55:18.598155 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3179bdb7-4679-4d55-a0f3-0b90b41414c3 None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:55:18.714360 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:55:18.731936 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3179bdb7-4679-4d55-a0f3-0b90b41414c3 None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:55:18.767332 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb5fa175-c338-4331-81bd-74fd0d0c3b24 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:55:18.767962 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb5fa175-c338-4331-81bd-74fd0d0c3b24 None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:55:18.768236 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb5fa175-c338-4331-81bd-74fd0d0c3b24 None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:55:18.768501 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb5fa175-c338-4331-81bd-74fd0d0c3b24 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:55:18.775594 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb5fa175-c338-4331-81bd-74fd0d0c3b24 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:55:18.777011 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb5fa175-c338-4331-81bd-74fd0d0c3b24 None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:55:18.787920 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application [None req-bb5fa175-c338-4331-81bd-74fd0d0c3b24 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 13 15:55:18.787920 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:55:18.787920 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:55:18.787920 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:55:18.787920 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:55:18.787920 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:55:18.787920 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:55:18.787920 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:55:18.787920 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:55:18.787920 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:55:18.787920 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:55:18.787920 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:55:18.787920 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 13 15:55:18.787920 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 13 15:55:18.787920 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 13 15:55:18.789061 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 13 15:55:18.789061 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:55:18.789061 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:55:18.789061 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:55:18.789061 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:55:18.789061 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:55:18.789061 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:55:18.789061 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:55:18.789061 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application with Lock( Sep 13 15:55:18.789061 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:55:18.789061 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:55:18.789061 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:55:18.789061 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:55:18.789061 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:55:18.789061 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:55:18.789061 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:55:18.789061 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:55:18.789981 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 13 15:55:18.789981 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 13 15:55:18.789981 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 13 15:55:18.789981 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 13 15:55:18.789981 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 13 15:55:18.789981 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 13 15:55:18.789981 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 13 15:55:18.789981 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Sep 13 15:55:18.789981 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 543/1104] 10.222.0.50 () {56 vars in 1111 bytes} [Tue Sep 13 15:55:18 2022] GET /identity/v3/projects/service => generated 88 bytes in 192 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:55:18.794803 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-553ce2cd-8ebb-4c9a-ae61-3a25336d0097 None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:55:18.838009 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:55:18.894848 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-553ce2cd-8ebb-4c9a-ae61-3a25336d0097 None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:55:18.930791 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64538c53-caa0-4121-a91b-05fbaf9fb0d0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:55:18.931426 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64538c53-caa0-4121-a91b-05fbaf9fb0d0 None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:55:18.931966 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64538c53-caa0-4121-a91b-05fbaf9fb0d0 None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:55:18.932236 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64538c53-caa0-4121-a91b-05fbaf9fb0d0 None admin] PATH_INFO: `/v3/projects` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:55:18.932924 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64538c53-caa0-4121-a91b-05fbaf9fb0d0 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81437) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:55:18.933227 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64538c53-caa0-4121-a91b-05fbaf9fb0d0 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:55:18.934650 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64538c53-caa0-4121-a91b-05fbaf9fb0d0 None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:55:18.946923 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 562/1105] 10.222.0.50 () {56 vars in 1148 bytes} [Tue Sep 13 15:55:18 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:55:18.953025 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb5fa175-c338-4331-81bd-74fd0d0c3b24 None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:55:19.053210 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb5fa175-c338-4331-81bd-74fd0d0c3b24 None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:55:19.087749 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97ba6411-59bb-4ac5-ac5f-75bee10a4758 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:55:19.088354 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97ba6411-59bb-4ac5-ac5f-75bee10a4758 None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:55:19.088635 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97ba6411-59bb-4ac5-ac5f-75bee10a4758 None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:55:19.088909 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97ba6411-59bb-4ac5-ac5f-75bee10a4758 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:55:19.089626 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97ba6411-59bb-4ac5-ac5f-75bee10a4758 None admin] RBAC: Adding query filter params (role.id=20e2f11234344900a51fc76fe5b625f3, scope.project.id=5811e3b83a1e462e9320cad79a3ad791, user.id=4e039b46c5e3445bbb39904b00c5430e) {{(pid=81438) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:55:19.089936 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97ba6411-59bb-4ac5-ac5f-75bee10a4758 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:55:19.091406 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97ba6411-59bb-4ac5-ac5f-75bee10a4758 None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:55:19.097169 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 544/1106] 10.222.0.50 () {56 vars in 1374 bytes} [Tue Sep 13 15:55:18 2022] GET /identity/v3/role_assignments?user.id=4e039b46c5e3445bbb39904b00c5430e&scope.project.id=5811e3b83a1e462e9320cad79a3ad791&role.id=20e2f11234344900a51fc76fe5b625f3 => generated 608 bytes in 145 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:55:20.702462 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7465b7a8-cd2b-4c3c-9291-bc55affe758d None None] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:55:20.703021 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7465b7a8-cd2b-4c3c-9291-bc55affe758d None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:55:20.703021 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7465b7a8-cd2b-4c3c-9291-bc55affe758d None None] PATH_INFO: `/` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:55:20.704357 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 563/1107] 10.222.0.50 () {54 vars in 927 bytes} [Tue Sep 13 15:55:20 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:55:20.710167 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68edc70d-37fa-4308-900a-64bfcbcbd8ce None None] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:55:20.710615 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68edc70d-37fa-4308-900a-64bfcbcbd8ce None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:55:20.710708 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68edc70d-37fa-4308-900a-64bfcbcbd8ce None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:55:20.729147 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.auth.core [None req-68edc70d-37fa-4308-900a-64bfcbcbd8ce None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81438) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:55:20.730323 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-68edc70d-37fa-4308-900a-64bfcbcbd8ce None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:55:20.733033 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:55:20.794842 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 545/1108] 10.222.0.50 () {58 vars in 1024 bytes} [Tue Sep 13 15:55:20 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 13 15:55:21.066517 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7465b7a8-cd2b-4c3c-9291-bc55affe758d None None] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:55:21.162492 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:55:21.167886 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7465b7a8-cd2b-4c3c-9291-bc55affe758d None None] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:55:21.200741 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8797a063-dbcd-4565-ba05-3841b5ef011b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:55:21.201353 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8797a063-dbcd-4565-ba05-3841b5ef011b None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:55:21.201629 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8797a063-dbcd-4565-ba05-3841b5ef011b None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:55:21.201899 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8797a063-dbcd-4565-ba05-3841b5ef011b None admin] PATH_INFO: `/v3/domains/default` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:55:21.203022 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8797a063-dbcd-4565-ba05-3841b5ef011b None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:55:21.204453 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8797a063-dbcd-4565-ba05-3841b5ef011b None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:55:21.205997 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 564/1109] 10.222.0.50 () {56 vars in 1108 bytes} [Tue Sep 13 15:55:21 2022] GET /identity/v3/domains/default => generated 199 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:55:21.212504 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68edc70d-37fa-4308-900a-64bfcbcbd8ce None None] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:55:21.307486 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68edc70d-37fa-4308-900a-64bfcbcbd8ce None None] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:55:21.339964 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3820561-07c1-4c1e-9bdc-6aaa3bfe4b07 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:55:21.340555 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3820561-07c1-4c1e-9bdc-6aaa3bfe4b07 None admin] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:55:21.340832 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3820561-07c1-4c1e-9bdc-6aaa3bfe4b07 None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:55:21.341104 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3820561-07c1-4c1e-9bdc-6aaa3bfe4b07 None admin] PATH_INFO: `/v3/users` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:55:21.341737 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3820561-07c1-4c1e-9bdc-6aaa3bfe4b07 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:55:21.343183 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3820561-07c1-4c1e-9bdc-6aaa3bfe4b07 None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:55:21.353982 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:55:21.365985 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 546/1110] 10.222.0.50 () {60 vars in 1132 bytes} [Tue Sep 13 15:55:21 2022] POST /identity/v3/users => generated 286 bytes in 154 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 15:55:23.547605 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b3b6baa-ecfa-4bc6-a4cc-29766df88297 None None] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:55:23.548133 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b3b6baa-ecfa-4bc6-a4cc-29766df88297 None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:55:23.548133 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b3b6baa-ecfa-4bc6-a4cc-29766df88297 None None] PATH_INFO: `/` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:55:23.549486 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 565/1111] 10.222.0.50 () {54 vars in 927 bytes} [Tue Sep 13 15:55:23 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:55:23.554889 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78ea51eb-e588-432b-9832-0394a634b8ff None None] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:55:23.555321 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78ea51eb-e588-432b-9832-0394a634b8ff None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:55:23.555465 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78ea51eb-e588-432b-9832-0394a634b8ff None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:55:23.574260 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.auth.core [None req-78ea51eb-e588-432b-9832-0394a634b8ff None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81438) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:55:23.575457 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-78ea51eb-e588-432b-9832-0394a634b8ff None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:55:23.640242 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 547/1112] 10.222.0.50 () {58 vars in 1024 bytes} [Tue Sep 13 15:55:23 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 13 15:55:23.915054 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b3b6baa-ecfa-4bc6-a4cc-29766df88297 None None] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:55:24.014382 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b3b6baa-ecfa-4bc6-a4cc-29766df88297 None None] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:55:24.048082 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea9fad80-70fd-4c5e-ac77-b4abc9f9cf9b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:55:24.048699 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea9fad80-70fd-4c5e-ac77-b4abc9f9cf9b None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:55:24.048974 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea9fad80-70fd-4c5e-ac77-b4abc9f9cf9b None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:55:24.049240 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea9fad80-70fd-4c5e-ac77-b4abc9f9cf9b None admin] PATH_INFO: `/v3/roles/f74e3f6c20bb426ca030f211c3a9aef5` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:55:24.050611 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea9fad80-70fd-4c5e-ac77-b4abc9f9cf9b None admin] RBAC: Authorizing `identity:get_role(role_id=f74e3f6c20bb426ca030f211c3a9aef5)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:55:24.052006 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea9fad80-70fd-4c5e-ac77-b4abc9f9cf9b None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:55:24.054394 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 566/1113] 10.222.0.50 () {56 vars in 1177 bytes} [Tue Sep 13 15:55:23 2022] GET /identity/v3/roles/f74e3f6c20bb426ca030f211c3a9aef5 => generated 223 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:55:24.060547 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78ea51eb-e588-432b-9832-0394a634b8ff None None] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:55:24.159933 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78ea51eb-e588-432b-9832-0394a634b8ff None None] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:55:24.186309 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:55:24.195653 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87f64d0a-2933-47c7-86e0-fd63be29a242 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:55:24.196275 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87f64d0a-2933-47c7-86e0-fd63be29a242 None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:55:24.196553 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87f64d0a-2933-47c7-86e0-fd63be29a242 None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:55:24.196821 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87f64d0a-2933-47c7-86e0-fd63be29a242 None admin] PATH_INFO: `/v3/users/4364e476404e410fa57b9f6f9b862246` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:55:24.210990 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87f64d0a-2933-47c7-86e0-fd63be29a242 None admin] RBAC: Authorizing `identity:get_user(user_id=4364e476404e410fa57b9f6f9b862246)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:55:24.212438 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87f64d0a-2933-47c7-86e0-fd63be29a242 None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:55:24.213967 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 548/1114] 10.222.0.50 () {56 vars in 1177 bytes} [Tue Sep 13 15:55:24 2022] GET /identity/v3/users/4364e476404e410fa57b9f6f9b862246 => generated 286 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:55:24.219710 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea9fad80-70fd-4c5e-ac77-b4abc9f9cf9b None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:55:24.260367 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:55:24.323914 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea9fad80-70fd-4c5e-ac77-b4abc9f9cf9b None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:55:24.357068 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8be8f1e1-a82f-47d4-a28d-a2c521856b2e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:55:24.357656 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8be8f1e1-a82f-47d4-a28d-a2c521856b2e None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:55:24.357933 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8be8f1e1-a82f-47d4-a28d-a2c521856b2e None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:55:24.358205 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8be8f1e1-a82f-47d4-a28d-a2c521856b2e None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:55:24.422701 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8be8f1e1-a82f-47d4-a28d-a2c521856b2e None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:55:24.424161 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8be8f1e1-a82f-47d4-a28d-a2c521856b2e None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:55:24.456020 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:55:24.518325 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 567/1115] 10.222.0.50 () {58 vars in 1282 bytes} [Tue Sep 13 15:55:24 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 299 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 13 15:55:24.525358 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87f64d0a-2933-47c7-86e0-fd63be29a242 None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:55:24.622859 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87f64d0a-2933-47c7-86e0-fd63be29a242 None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:55:24.656288 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2706fd3c-6126-41e1-87d7-7c9492ed844b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:55:24.656904 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2706fd3c-6126-41e1-87d7-7c9492ed844b None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:55:24.657182 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2706fd3c-6126-41e1-87d7-7c9492ed844b None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:55:24.657450 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2706fd3c-6126-41e1-87d7-7c9492ed844b None admin] PATH_INFO: `/v3/projects/13d63282b1fd48349397eed6fccbd62a` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:55:24.658533 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2706fd3c-6126-41e1-87d7-7c9492ed844b None admin] RBAC: Authorizing `identity:get_project(project_id=13d63282b1fd48349397eed6fccbd62a)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:55:24.659968 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2706fd3c-6126-41e1-87d7-7c9492ed844b None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:55:24.662224 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 549/1116] 10.222.0.50 () {56 vars in 1186 bytes} [Tue Sep 13 15:55:24 2022] GET /identity/v3/projects/13d63282b1fd48349397eed6fccbd62a => generated 311 bytes in 137 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:55:24.668216 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8be8f1e1-a82f-47d4-a28d-a2c521856b2e None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:55:24.764869 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8be8f1e1-a82f-47d4-a28d-a2c521856b2e None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:55:24.797497 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-082f87ac-c5d4-40b1-8538-4d098dd3663b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:55:24.798085 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-082f87ac-c5d4-40b1-8538-4d098dd3663b None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:55:24.798362 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-082f87ac-c5d4-40b1-8538-4d098dd3663b None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:55:24.798633 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-082f87ac-c5d4-40b1-8538-4d098dd3663b None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:55:24.799372 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-082f87ac-c5d4-40b1-8538-4d098dd3663b None admin] RBAC: Adding query filter params (role.id=f74e3f6c20bb426ca030f211c3a9aef5, scope.project.id=13d63282b1fd48349397eed6fccbd62a, user.id=4364e476404e410fa57b9f6f9b862246) {{(pid=81437) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:55:24.799679 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-082f87ac-c5d4-40b1-8538-4d098dd3663b None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:55:24.801092 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-082f87ac-c5d4-40b1-8538-4d098dd3663b None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:55:24.804389 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:55:24.807709 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 568/1117] 10.222.0.50 () {56 vars in 1374 bytes} [Tue Sep 13 15:55:24 2022] GET /identity/v3/role_assignments?user.id=4364e476404e410fa57b9f6f9b862246&scope.project.id=13d63282b1fd48349397eed6fccbd62a&role.id=f74e3f6c20bb426ca030f211c3a9aef5 => generated 259 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:55:26.961709 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb9439b0-c23d-4838-8e66-35fbb495403e None None] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:55:26.962270 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb9439b0-c23d-4838-8e66-35fbb495403e None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:55:26.962270 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb9439b0-c23d-4838-8e66-35fbb495403e None None] PATH_INFO: `/` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:55:26.963592 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 550/1118] 10.222.0.50 () {54 vars in 927 bytes} [Tue Sep 13 15:55:26 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:55:26.968803 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b098a5c3-7cf6-43d4-a58e-4694707f7d74 None None] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:55:26.969211 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b098a5c3-7cf6-43d4-a58e-4694707f7d74 None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:55:26.969349 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b098a5c3-7cf6-43d4-a58e-4694707f7d74 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:55:26.987383 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.auth.core [None req-b098a5c3-7cf6-43d4-a58e-4694707f7d74 None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81437) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:55:26.988556 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-b098a5c3-7cf6-43d4-a58e-4694707f7d74 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:55:27.055860 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 569/1119] 10.222.0.50 () {58 vars in 1024 bytes} [Tue Sep 13 15:55:26 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 13 15:55:27.326700 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb9439b0-c23d-4838-8e66-35fbb495403e None None] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:55:27.378000 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:55:27.423336 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb9439b0-c23d-4838-8e66-35fbb495403e None None] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:55:27.456106 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b991461e-a45e-42f2-868a-b525ae7cc7c2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:55:27.456720 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b991461e-a45e-42f2-868a-b525ae7cc7c2 None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:55:27.456995 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b991461e-a45e-42f2-868a-b525ae7cc7c2 None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:55:27.457263 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b991461e-a45e-42f2-868a-b525ae7cc7c2 None admin] PATH_INFO: `/v3/roles/f74e3f6c20bb426ca030f211c3a9aef5` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:55:27.458767 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b991461e-a45e-42f2-868a-b525ae7cc7c2 None admin] RBAC: Authorizing `identity:get_role(role_id=f74e3f6c20bb426ca030f211c3a9aef5)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:55:27.460159 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b991461e-a45e-42f2-868a-b525ae7cc7c2 None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:55:27.461441 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 551/1120] 10.222.0.50 () {56 vars in 1177 bytes} [Tue Sep 13 15:55:27 2022] GET /identity/v3/roles/f74e3f6c20bb426ca030f211c3a9aef5 => generated 223 bytes in 135 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:55:27.467301 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b098a5c3-7cf6-43d4-a58e-4694707f7d74 None None] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:55:27.563054 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b098a5c3-7cf6-43d4-a58e-4694707f7d74 None None] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:55:27.590835 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:55:27.597146 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e96980bb-1785-4e03-9a40-1f7b80049394 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:55:27.597767 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e96980bb-1785-4e03-9a40-1f7b80049394 None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:55:27.598045 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e96980bb-1785-4e03-9a40-1f7b80049394 None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:55:27.598316 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e96980bb-1785-4e03-9a40-1f7b80049394 None admin] PATH_INFO: `/v3/users/4364e476404e410fa57b9f6f9b862246` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:55:27.600108 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e96980bb-1785-4e03-9a40-1f7b80049394 None admin] RBAC: Authorizing `identity:get_user(user_id=4364e476404e410fa57b9f6f9b862246)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:55:27.601538 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e96980bb-1785-4e03-9a40-1f7b80049394 None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:55:27.603128 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 570/1121] 10.222.0.50 () {56 vars in 1177 bytes} [Tue Sep 13 15:55:27 2022] GET /identity/v3/users/4364e476404e410fa57b9f6f9b862246 => generated 286 bytes in 136 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:55:27.608938 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b991461e-a45e-42f2-868a-b525ae7cc7c2 None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:55:27.707398 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b991461e-a45e-42f2-868a-b525ae7cc7c2 None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:55:27.727777 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:55:27.742364 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-656160b2-fa9d-4845-84a1-0cefe4fb7bfa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:55:27.743079 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-656160b2-fa9d-4845-84a1-0cefe4fb7bfa None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:55:27.743363 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-656160b2-fa9d-4845-84a1-0cefe4fb7bfa None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:55:27.743639 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-656160b2-fa9d-4845-84a1-0cefe4fb7bfa None admin] PATH_INFO: `/v3/projects/13d63282b1fd48349397eed6fccbd62a` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:55:27.744747 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-656160b2-fa9d-4845-84a1-0cefe4fb7bfa None admin] RBAC: Authorizing `identity:get_project(project_id=13d63282b1fd48349397eed6fccbd62a)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:55:27.746172 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-656160b2-fa9d-4845-84a1-0cefe4fb7bfa None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:55:27.747827 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 552/1122] 10.222.0.50 () {56 vars in 1186 bytes} [Tue Sep 13 15:55:27 2022] GET /identity/v3/projects/13d63282b1fd48349397eed6fccbd62a => generated 311 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:55:27.753770 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e96980bb-1785-4e03-9a40-1f7b80049394 None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:55:27.849995 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e96980bb-1785-4e03-9a40-1f7b80049394 None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:55:27.882503 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04f568a4-7687-41cb-815a-5ef09c2aa17f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:55:27.883167 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04f568a4-7687-41cb-815a-5ef09c2aa17f None admin] REQUEST_METHOD: `PUT` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:55:27.883448 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04f568a4-7687-41cb-815a-5ef09c2aa17f None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:55:27.883724 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04f568a4-7687-41cb-815a-5ef09c2aa17f None admin] PATH_INFO: `/v3/projects/13d63282b1fd48349397eed6fccbd62a/users/4364e476404e410fa57b9f6f9b862246/roles/f74e3f6c20bb426ca030f211c3a9aef5` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:55:27.885816 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04f568a4-7687-41cb-815a-5ef09c2aa17f None admin] RBAC: Authorizing `identity:create_grant(project_id=13d63282b1fd48349397eed6fccbd62a, user_id=4364e476404e410fa57b9f6f9b862246, role_id=f74e3f6c20bb426ca030f211c3a9aef5)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:55:27.887275 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04f568a4-7687-41cb-815a-5ef09c2aa17f None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:55:27.899489 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 571/1123] 10.222.0.50 () {58 vars in 1439 bytes} [Tue Sep 13 15:55:27 2022] PUT /identity/v3/projects/13d63282b1fd48349397eed6fccbd62a/users/4364e476404e410fa57b9f6f9b862246/roles/f74e3f6c20bb426ca030f211c3a9aef5 => generated 0 bytes in 146 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:55:30.011235 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10d49601-4e7e-4be4-af08-15677dd5f62d None None] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:55:30.011788 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10d49601-4e7e-4be4-af08-15677dd5f62d None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:55:30.011788 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10d49601-4e7e-4be4-af08-15677dd5f62d None None] PATH_INFO: `/` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:55:30.013128 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 553/1124] 10.222.0.50 () {54 vars in 927 bytes} [Tue Sep 13 15:55:30 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:55:30.018440 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49533453-1d5c-4361-9556-1ef65c1e1067 None None] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:55:30.018858 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49533453-1d5c-4361-9556-1ef65c1e1067 None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:55:30.019021 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49533453-1d5c-4361-9556-1ef65c1e1067 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:55:30.037372 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.auth.core [None req-49533453-1d5c-4361-9556-1ef65c1e1067 None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81437) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:55:30.038547 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-49533453-1d5c-4361-9556-1ef65c1e1067 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:55:30.090284 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:55:30.104575 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 572/1125] 10.222.0.50 () {58 vars in 1024 bytes} [Tue Sep 13 15:55:30 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 13 15:55:30.376667 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10d49601-4e7e-4be4-af08-15677dd5f62d None None] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:55:30.474976 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10d49601-4e7e-4be4-af08-15677dd5f62d None None] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:55:30.507148 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64fc5874-d9c5-47a6-a066-99a171d7d839 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:55:30.507770 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64fc5874-d9c5-47a6-a066-99a171d7d839 None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:55:30.508055 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64fc5874-d9c5-47a6-a066-99a171d7d839 None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:55:30.508331 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64fc5874-d9c5-47a6-a066-99a171d7d839 None admin] PATH_INFO: `/v3/roles/f74e3f6c20bb426ca030f211c3a9aef5` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:55:30.509717 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64fc5874-d9c5-47a6-a066-99a171d7d839 None admin] RBAC: Authorizing `identity:get_role(role_id=f74e3f6c20bb426ca030f211c3a9aef5)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:55:30.511111 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64fc5874-d9c5-47a6-a066-99a171d7d839 None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:55:30.512419 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 554/1126] 10.222.0.50 () {56 vars in 1177 bytes} [Tue Sep 13 15:55:30 2022] GET /identity/v3/roles/f74e3f6c20bb426ca030f211c3a9aef5 => generated 223 bytes in 136 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:55:30.518333 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49533453-1d5c-4361-9556-1ef65c1e1067 None None] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:55:30.615548 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49533453-1d5c-4361-9556-1ef65c1e1067 None None] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:55:30.649229 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-870782e7-303d-4a36-a67a-946d60fc580d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:55:30.649783 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-870782e7-303d-4a36-a67a-946d60fc580d None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:55:30.650063 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-870782e7-303d-4a36-a67a-946d60fc580d None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:55:30.650338 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-870782e7-303d-4a36-a67a-946d60fc580d None admin] PATH_INFO: `/v3/users/4364e476404e410fa57b9f6f9b862246` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:55:30.651468 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-870782e7-303d-4a36-a67a-946d60fc580d None admin] RBAC: Authorizing `identity:get_user(user_id=4364e476404e410fa57b9f6f9b862246)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:55:30.652905 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-870782e7-303d-4a36-a67a-946d60fc580d None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:55:30.654452 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 573/1127] 10.222.0.50 () {56 vars in 1177 bytes} [Tue Sep 13 15:55:30 2022] GET /identity/v3/users/4364e476404e410fa57b9f6f9b862246 => generated 286 bytes in 137 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:55:30.660147 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64fc5874-d9c5-47a6-a066-99a171d7d839 None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:55:30.714803 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:55:30.756596 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64fc5874-d9c5-47a6-a066-99a171d7d839 None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:55:30.789497 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e80a03af-2d8f-420f-8753-22edc17db9dc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:55:30.790095 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e80a03af-2d8f-420f-8753-22edc17db9dc None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:55:30.790377 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e80a03af-2d8f-420f-8753-22edc17db9dc None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:55:30.790652 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e80a03af-2d8f-420f-8753-22edc17db9dc None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:55:30.855822 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e80a03af-2d8f-420f-8753-22edc17db9dc None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:55:30.857257 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e80a03af-2d8f-420f-8753-22edc17db9dc None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:55:30.910308 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:55:30.952922 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 555/1128] 10.222.0.50 () {58 vars in 1282 bytes} [Tue Sep 13 15:55:30 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 293 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 13 15:55:30.959534 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-870782e7-303d-4a36-a67a-946d60fc580d None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:55:31.020439 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:55:31.064647 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-870782e7-303d-4a36-a67a-946d60fc580d None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:55:31.097360 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed628e4d-c5df-4afa-b05e-1216c6729e49 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:55:31.097980 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed628e4d-c5df-4afa-b05e-1216c6729e49 None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:55:31.098258 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed628e4d-c5df-4afa-b05e-1216c6729e49 None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:55:31.098591 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed628e4d-c5df-4afa-b05e-1216c6729e49 None admin] PATH_INFO: `/v3/projects/13d63282b1fd48349397eed6fccbd62a` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:55:31.099800 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed628e4d-c5df-4afa-b05e-1216c6729e49 None admin] RBAC: Authorizing `identity:get_project(project_id=13d63282b1fd48349397eed6fccbd62a)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:55:31.101284 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed628e4d-c5df-4afa-b05e-1216c6729e49 None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:55:31.102905 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 574/1129] 10.222.0.50 () {56 vars in 1186 bytes} [Tue Sep 13 15:55:30 2022] GET /identity/v3/projects/13d63282b1fd48349397eed6fccbd62a => generated 311 bytes in 145 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:55:31.109097 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e80a03af-2d8f-420f-8753-22edc17db9dc None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:55:31.205487 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e80a03af-2d8f-420f-8753-22edc17db9dc None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:55:31.237505 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a086c7a1-6b6c-45cf-a804-9772853cd15b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:55:31.238098 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a086c7a1-6b6c-45cf-a804-9772853cd15b None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:55:31.238382 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a086c7a1-6b6c-45cf-a804-9772853cd15b None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:55:31.238656 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a086c7a1-6b6c-45cf-a804-9772853cd15b None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:55:31.240159 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a086c7a1-6b6c-45cf-a804-9772853cd15b None admin] RBAC: Adding query filter params (role.id=f74e3f6c20bb426ca030f211c3a9aef5, scope.project.id=13d63282b1fd48349397eed6fccbd62a, user.id=4364e476404e410fa57b9f6f9b862246) {{(pid=81438) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:55:31.240509 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a086c7a1-6b6c-45cf-a804-9772853cd15b None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:55:31.241923 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a086c7a1-6b6c-45cf-a804-9772853cd15b None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:55:31.247464 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 556/1130] 10.222.0.50 () {56 vars in 1374 bytes} [Tue Sep 13 15:55:31 2022] GET /identity/v3/role_assignments?user.id=4364e476404e410fa57b9f6f9b862246&scope.project.id=13d63282b1fd48349397eed6fccbd62a&role.id=f74e3f6c20bb426ca030f211c3a9aef5 => generated 608 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:55:33.401474 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04d90e82-10a8-4b3f-95a3-5417666ec756 None None] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:55:33.402047 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04d90e82-10a8-4b3f-95a3-5417666ec756 None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:55:33.402047 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04d90e82-10a8-4b3f-95a3-5417666ec756 None None] PATH_INFO: `/` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:55:33.403399 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 575/1131] 10.222.0.50 () {54 vars in 927 bytes} [Tue Sep 13 15:55:33 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:55:33.408577 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f81a266-7794-496d-8962-b5ad2af864ce None None] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:55:33.408958 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f81a266-7794-496d-8962-b5ad2af864ce None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:55:33.409115 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f81a266-7794-496d-8962-b5ad2af864ce None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:55:33.428111 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.auth.core [None req-8f81a266-7794-496d-8962-b5ad2af864ce None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81438) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:55:33.429281 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-8f81a266-7794-496d-8962-b5ad2af864ce None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:55:33.443457 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:55:33.495573 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 557/1132] 10.222.0.50 () {58 vars in 1024 bytes} [Tue Sep 13 15:55:33 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 13 15:55:33.764055 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04d90e82-10a8-4b3f-95a3-5417666ec756 None None] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:55:33.863442 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04d90e82-10a8-4b3f-95a3-5417666ec756 None None] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:55:33.886337 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:55:33.897460 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ef5cc02-0c0c-41f6-bda2-6021eef1e64e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:55:33.898078 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ef5cc02-0c0c-41f6-bda2-6021eef1e64e None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:55:33.898359 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ef5cc02-0c0c-41f6-bda2-6021eef1e64e None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:55:33.898632 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ef5cc02-0c0c-41f6-bda2-6021eef1e64e None admin] PATH_INFO: `/v3/domains/default` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:55:33.899771 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ef5cc02-0c0c-41f6-bda2-6021eef1e64e None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:55:33.901204 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ef5cc02-0c0c-41f6-bda2-6021eef1e64e None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:55:33.902747 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 576/1133] 10.222.0.50 () {56 vars in 1108 bytes} [Tue Sep 13 15:55:33 2022] GET /identity/v3/domains/default => generated 199 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:55:33.908513 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f81a266-7794-496d-8962-b5ad2af864ce None None] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:55:34.004032 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f81a266-7794-496d-8962-b5ad2af864ce None None] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:55:34.037110 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28158c8d-29a5-473b-aa1e-9e3324329a22 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:55:34.037642 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28158c8d-29a5-473b-aa1e-9e3324329a22 None admin] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:55:34.037918 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28158c8d-29a5-473b-aa1e-9e3324329a22 None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:55:34.038189 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28158c8d-29a5-473b-aa1e-9e3324329a22 None admin] PATH_INFO: `/v3/projects` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:55:34.038833 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28158c8d-29a5-473b-aa1e-9e3324329a22 None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:55:34.040297 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28158c8d-29a5-473b-aa1e-9e3324329a22 None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:55:34.066273 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 558/1134] 10.222.0.50 () {60 vars in 1141 bytes} [Tue Sep 13 15:55:33 2022] POST /identity/v3/projects => generated 311 bytes in 158 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 15:55:36.220857 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9968fc9e-a959-44b8-ab8d-efe7c9803bbe None None] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:55:36.221378 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9968fc9e-a959-44b8-ab8d-efe7c9803bbe None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:55:36.221378 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9968fc9e-a959-44b8-ab8d-efe7c9803bbe None None] PATH_INFO: `/` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:55:36.222716 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 577/1135] 10.222.0.50 () {54 vars in 927 bytes} [Tue Sep 13 15:55:36 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:55:36.228661 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75ed1e6a-511d-4454-addb-2e31942d4c92 None None] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:55:36.229072 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75ed1e6a-511d-4454-addb-2e31942d4c92 None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:55:36.229211 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75ed1e6a-511d-4454-addb-2e31942d4c92 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:55:36.235432 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:55:36.248989 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.auth.core [None req-75ed1e6a-511d-4454-addb-2e31942d4c92 None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81438) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:55:36.250170 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-75ed1e6a-511d-4454-addb-2e31942d4c92 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:55:36.315766 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 559/1136] 10.222.0.50 () {58 vars in 1024 bytes} [Tue Sep 13 15:55:36 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 13 15:55:36.585147 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9968fc9e-a959-44b8-ab8d-efe7c9803bbe None None] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:55:36.683674 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9968fc9e-a959-44b8-ab8d-efe7c9803bbe None None] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:55:36.717780 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-718bfc31-c016-4c02-98d8-6198cc52c537 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:55:36.718394 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-718bfc31-c016-4c02-98d8-6198cc52c537 None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:55:36.718670 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-718bfc31-c016-4c02-98d8-6198cc52c537 None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:55:36.718963 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-718bfc31-c016-4c02-98d8-6198cc52c537 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:55:36.720075 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-718bfc31-c016-4c02-98d8-6198cc52c537 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:55:36.721494 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-718bfc31-c016-4c02-98d8-6198cc52c537 None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:55:36.723045 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 578/1137] 10.222.0.50 () {56 vars in 1108 bytes} [Tue Sep 13 15:55:36 2022] GET /identity/v3/domains/default => generated 199 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:55:36.728923 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75ed1e6a-511d-4454-addb-2e31942d4c92 None None] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:55:36.823453 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75ed1e6a-511d-4454-addb-2e31942d4c92 None None] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:55:36.851729 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:55:36.856753 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17e7e9cc-48e4-438c-a7ef-4618317dec27 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:55:36.857346 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17e7e9cc-48e4-438c-a7ef-4618317dec27 None admin] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:55:36.857619 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17e7e9cc-48e4-438c-a7ef-4618317dec27 None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:55:36.857886 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17e7e9cc-48e4-438c-a7ef-4618317dec27 None admin] PATH_INFO: `/v3/users` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:55:36.858527 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17e7e9cc-48e4-438c-a7ef-4618317dec27 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:55:36.859970 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17e7e9cc-48e4-438c-a7ef-4618317dec27 None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:55:36.880435 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 560/1138] 10.222.0.50 () {60 vars in 1132 bytes} [Tue Sep 13 15:55:36 2022] POST /identity/v3/users => generated 286 bytes in 152 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 15:55:39.060512 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec34b88a-4411-4f0e-ac5c-5b098e4ee258 None None] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:55:39.061063 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec34b88a-4411-4f0e-ac5c-5b098e4ee258 None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:55:39.061063 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec34b88a-4411-4f0e-ac5c-5b098e4ee258 None None] PATH_INFO: `/` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:55:39.062710 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 579/1139] 10.222.0.50 () {54 vars in 927 bytes} [Tue Sep 13 15:55:39 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:55:39.067815 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3aacdca8-7e16-4fff-9fdc-acb4152f17c4 None None] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:55:39.068199 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3aacdca8-7e16-4fff-9fdc-acb4152f17c4 None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:55:39.068375 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3aacdca8-7e16-4fff-9fdc-acb4152f17c4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:55:39.086874 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.auth.core [None req-3aacdca8-7e16-4fff-9fdc-acb4152f17c4 None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81438) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:55:39.088072 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-3aacdca8-7e16-4fff-9fdc-acb4152f17c4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:55:39.152920 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 561/1140] 10.222.0.50 () {58 vars in 1024 bytes} [Tue Sep 13 15:55:39 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 13 15:55:39.422062 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec34b88a-4411-4f0e-ac5c-5b098e4ee258 None None] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:55:39.481181 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:55:39.521882 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec34b88a-4411-4f0e-ac5c-5b098e4ee258 None None] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:55:39.554966 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e45f2275-fc3b-4835-b812-ba8e836e7882 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:55:39.555569 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e45f2275-fc3b-4835-b812-ba8e836e7882 None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:55:39.555844 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e45f2275-fc3b-4835-b812-ba8e836e7882 None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:55:39.556375 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e45f2275-fc3b-4835-b812-ba8e836e7882 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:55:39.566103 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e45f2275-fc3b-4835-b812-ba8e836e7882 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 13 15:55:39.571623 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application [None req-e45f2275-fc3b-4835-b812-ba8e836e7882 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 13 15:55:39.571623 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:55:39.571623 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:55:39.571623 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:55:39.571623 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:55:39.571623 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:55:39.571623 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:55:39.571623 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:55:39.571623 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:55:39.571623 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:55:39.571623 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:55:39.571623 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:55:39.571623 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 13 15:55:39.571623 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 13 15:55:39.571623 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 13 15:55:39.571623 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 13 15:55:39.572738 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 13 15:55:39.572738 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 13 15:55:39.572738 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 13 15:55:39.572738 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 13 15:55:39.572738 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:55:39.572738 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:55:39.572738 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:55:39.572738 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:55:39.572738 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:55:39.572738 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:55:39.572738 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:55:39.572738 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application with Lock( Sep 13 15:55:39.572738 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:55:39.572738 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:55:39.572738 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:55:39.572738 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:55:39.574009 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:55:39.574009 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:55:39.574009 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:55:39.574009 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:55:39.574009 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 13 15:55:39.574009 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 13 15:55:39.574009 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 13 15:55:39.574009 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 13 15:55:39.574009 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 13 15:55:39.574009 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 13 15:55:39.574009 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 13 15:55:39.574009 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application Sep 13 15:55:39.574009 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 580/1141] 10.222.0.50 () {56 vars in 1096 bytes} [Tue Sep 13 15:55:39 2022] GET /identity/v3/roles/admin => generated 83 bytes in 151 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:55:39.578675 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3aacdca8-7e16-4fff-9fdc-acb4152f17c4 None None] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:55:39.673669 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3aacdca8-7e16-4fff-9fdc-acb4152f17c4 None None] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:55:39.690187 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:55:39.708321 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d4d85a0-d64c-4024-9909-03fe10676c72 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:55:39.708917 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d4d85a0-d64c-4024-9909-03fe10676c72 None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:55:39.709197 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d4d85a0-d64c-4024-9909-03fe10676c72 None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:55:39.709470 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d4d85a0-d64c-4024-9909-03fe10676c72 None admin] PATH_INFO: `/v3/roles` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:55:39.710122 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d4d85a0-d64c-4024-9909-03fe10676c72 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81438) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:55:39.710462 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d4d85a0-d64c-4024-9909-03fe10676c72 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:55:39.712007 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d4d85a0-d64c-4024-9909-03fe10676c72 None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:55:39.720022 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 562/1142] 10.222.0.50 () {56 vars in 1099 bytes} [Tue Sep 13 15:55:39 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 142 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:55:39.726056 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e45f2275-fc3b-4835-b812-ba8e836e7882 None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:55:39.822605 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e45f2275-fc3b-4835-b812-ba8e836e7882 None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:55:39.842138 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:55:39.855909 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcfb1488-322e-466e-bb4a-c425c4e8a9e6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:55:39.856520 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcfb1488-322e-466e-bb4a-c425c4e8a9e6 None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:55:39.856794 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcfb1488-322e-466e-bb4a-c425c4e8a9e6 None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:55:39.857064 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcfb1488-322e-466e-bb4a-c425c4e8a9e6 None admin] PATH_INFO: `/v3/users/b2bf9837adc84a3c8a756f161413468f` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:55:39.870287 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fcfb1488-322e-466e-bb4a-c425c4e8a9e6 None admin] RBAC: Authorizing `identity:get_user(user_id=b2bf9837adc84a3c8a756f161413468f)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:55:39.871736 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fcfb1488-322e-466e-bb4a-c425c4e8a9e6 None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:55:39.873299 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 581/1143] 10.222.0.50 () {56 vars in 1177 bytes} [Tue Sep 13 15:55:39 2022] GET /identity/v3/users/b2bf9837adc84a3c8a756f161413468f => generated 286 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:55:39.879683 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d4d85a0-d64c-4024-9909-03fe10676c72 None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:55:39.976602 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d4d85a0-d64c-4024-9909-03fe10676c72 None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:55:40.008822 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bedb61fd-ba88-4672-8765-5172d0562457 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:55:40.009420 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bedb61fd-ba88-4672-8765-5172d0562457 None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:55:40.009699 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bedb61fd-ba88-4672-8765-5172d0562457 None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:55:40.009970 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bedb61fd-ba88-4672-8765-5172d0562457 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:55:40.056829 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:55:40.075953 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bedb61fd-ba88-4672-8765-5172d0562457 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:55:40.077383 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bedb61fd-ba88-4672-8765-5172d0562457 None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:55:40.172370 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 563/1144] 10.222.0.50 () {58 vars in 1282 bytes} [Tue Sep 13 15:55:39 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 294 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 13 15:55:40.178654 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcfb1488-322e-466e-bb4a-c425c4e8a9e6 None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:55:40.274047 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcfb1488-322e-466e-bb4a-c425c4e8a9e6 None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:55:40.306675 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fc1380f-1c76-4c88-8b3d-553bde411763 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:55:40.307312 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fc1380f-1c76-4c88-8b3d-553bde411763 None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:55:40.307588 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fc1380f-1c76-4c88-8b3d-553bde411763 None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:55:40.307858 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fc1380f-1c76-4c88-8b3d-553bde411763 None admin] PATH_INFO: `/v3/projects/4cad029b0df84051812bcec923a7adc6` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:55:40.308937 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fc1380f-1c76-4c88-8b3d-553bde411763 None admin] RBAC: Authorizing `identity:get_project(project_id=4cad029b0df84051812bcec923a7adc6)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:55:40.310344 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fc1380f-1c76-4c88-8b3d-553bde411763 None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:55:40.311959 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 582/1145] 10.222.0.50 () {56 vars in 1186 bytes} [Tue Sep 13 15:55:40 2022] GET /identity/v3/projects/4cad029b0df84051812bcec923a7adc6 => generated 311 bytes in 134 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:55:40.318838 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bedb61fd-ba88-4672-8765-5172d0562457 None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:55:40.397273 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:55:40.415937 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bedb61fd-ba88-4672-8765-5172d0562457 None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:55:40.448956 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-547d0b83-2101-490a-9241-14aeb96b25eb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:55:40.449550 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-547d0b83-2101-490a-9241-14aeb96b25eb None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:55:40.449829 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-547d0b83-2101-490a-9241-14aeb96b25eb None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:55:40.450102 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-547d0b83-2101-490a-9241-14aeb96b25eb None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:55:40.450813 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-547d0b83-2101-490a-9241-14aeb96b25eb None admin] RBAC: Adding query filter params (role.id=031d4e1a23e34f71858fc4a6dfbd4ccb, scope.project.id=4cad029b0df84051812bcec923a7adc6, user.id=b2bf9837adc84a3c8a756f161413468f) {{(pid=81438) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:55:40.451150 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-547d0b83-2101-490a-9241-14aeb96b25eb None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:55:40.452560 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-547d0b83-2101-490a-9241-14aeb96b25eb None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:55:40.457843 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 564/1146] 10.222.0.50 () {56 vars in 1374 bytes} [Tue Sep 13 15:55:40 2022] GET /identity/v3/role_assignments?user.id=b2bf9837adc84a3c8a756f161413468f&scope.project.id=4cad029b0df84051812bcec923a7adc6&role.id=031d4e1a23e34f71858fc4a6dfbd4ccb => generated 259 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:55:42.577780 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e644fa59-008b-49bf-a75e-ef5e6cf4d281 None None] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:55:42.578337 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e644fa59-008b-49bf-a75e-ef5e6cf4d281 None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:55:42.578337 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e644fa59-008b-49bf-a75e-ef5e6cf4d281 None None] PATH_INFO: `/` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:55:42.579677 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 583/1147] 10.222.0.50 () {54 vars in 927 bytes} [Tue Sep 13 15:55:42 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:55:42.585574 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cbf1c31-bd89-43bd-8cd6-083c205acaf6 None None] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:55:42.585955 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cbf1c31-bd89-43bd-8cd6-083c205acaf6 None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:55:42.586115 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cbf1c31-bd89-43bd-8cd6-083c205acaf6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:55:42.604559 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.auth.core [None req-2cbf1c31-bd89-43bd-8cd6-083c205acaf6 None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81438) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:55:42.605740 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-2cbf1c31-bd89-43bd-8cd6-083c205acaf6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:55:42.672070 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 565/1148] 10.222.0.50 () {58 vars in 1024 bytes} [Tue Sep 13 15:55:42 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 13 15:55:42.942467 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e644fa59-008b-49bf-a75e-ef5e6cf4d281 None None] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:55:42.989340 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:55:43.041938 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e644fa59-008b-49bf-a75e-ef5e6cf4d281 None None] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:55:43.078071 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a75613aa-36ee-4d5e-b721-8eaa60caa111 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:55:43.078696 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a75613aa-36ee-4d5e-b721-8eaa60caa111 None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:55:43.079009 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a75613aa-36ee-4d5e-b721-8eaa60caa111 None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:55:43.079282 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a75613aa-36ee-4d5e-b721-8eaa60caa111 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:55:43.089179 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a75613aa-36ee-4d5e-b721-8eaa60caa111 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 13 15:55:43.093901 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application [None req-a75613aa-36ee-4d5e-b721-8eaa60caa111 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 13 15:55:43.093901 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:55:43.093901 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:55:43.093901 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:55:43.093901 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:55:43.093901 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:55:43.093901 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:55:43.093901 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:55:43.093901 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:55:43.093901 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:55:43.093901 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:55:43.093901 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:55:43.093901 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 13 15:55:43.093901 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 13 15:55:43.093901 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 13 15:55:43.093901 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 13 15:55:43.095027 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 13 15:55:43.095027 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 13 15:55:43.095027 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 13 15:55:43.095027 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 13 15:55:43.095027 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:55:43.095027 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:55:43.095027 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:55:43.095027 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:55:43.095027 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:55:43.095027 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:55:43.095027 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:55:43.095027 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application with Lock( Sep 13 15:55:43.095027 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:55:43.095027 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:55:43.095027 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:55:43.095027 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:55:43.096089 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:55:43.096089 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:55:43.096089 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:55:43.096089 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:55:43.096089 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 13 15:55:43.096089 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 13 15:55:43.096089 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 13 15:55:43.096089 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 13 15:55:43.096089 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 13 15:55:43.096089 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 13 15:55:43.096089 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 13 15:55:43.096089 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application Sep 13 15:55:43.096089 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 584/1149] 10.222.0.50 () {56 vars in 1096 bytes} [Tue Sep 13 15:55:42 2022] GET /identity/v3/roles/admin => generated 83 bytes in 154 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:55:43.101246 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cbf1c31-bd89-43bd-8cd6-083c205acaf6 None None] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:55:43.172763 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:55:43.197778 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cbf1c31-bd89-43bd-8cd6-083c205acaf6 None None] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:55:43.229473 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c575c026-e08d-43fe-9d20-07e44ac0767f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:55:43.230065 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c575c026-e08d-43fe-9d20-07e44ac0767f None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:55:43.230341 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c575c026-e08d-43fe-9d20-07e44ac0767f None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:55:43.230608 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c575c026-e08d-43fe-9d20-07e44ac0767f None admin] PATH_INFO: `/v3/roles` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:55:43.231281 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c575c026-e08d-43fe-9d20-07e44ac0767f None admin] RBAC: Adding query filter params (name=admin) {{(pid=81438) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:55:43.231578 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c575c026-e08d-43fe-9d20-07e44ac0767f None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:55:43.232918 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c575c026-e08d-43fe-9d20-07e44ac0767f None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:55:43.241139 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 566/1150] 10.222.0.50 () {56 vars in 1099 bytes} [Tue Sep 13 15:55:43 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 13 15:55:43.246914 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a75613aa-36ee-4d5e-b721-8eaa60caa111 None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:55:43.343808 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a75613aa-36ee-4d5e-b721-8eaa60caa111 None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:55:43.351534 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:55:43.378631 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19faaf97-af3f-47c3-90a4-d75959b116e2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:55:43.379327 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19faaf97-af3f-47c3-90a4-d75959b116e2 None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:55:43.379653 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19faaf97-af3f-47c3-90a4-d75959b116e2 None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:55:43.379929 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19faaf97-af3f-47c3-90a4-d75959b116e2 None admin] PATH_INFO: `/v3/users/b2bf9837adc84a3c8a756f161413468f` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:55:43.381044 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19faaf97-af3f-47c3-90a4-d75959b116e2 None admin] RBAC: Authorizing `identity:get_user(user_id=b2bf9837adc84a3c8a756f161413468f)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:55:43.382476 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19faaf97-af3f-47c3-90a4-d75959b116e2 None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:55:43.384078 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 585/1151] 10.222.0.50 () {56 vars in 1177 bytes} [Tue Sep 13 15:55:43 2022] GET /identity/v3/users/b2bf9837adc84a3c8a756f161413468f => generated 286 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:55:43.389773 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c575c026-e08d-43fe-9d20-07e44ac0767f None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:55:43.485598 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c575c026-e08d-43fe-9d20-07e44ac0767f None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:55:43.517860 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7554000-e152-42cb-ad12-490b79b6b6d1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:55:43.518471 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7554000-e152-42cb-ad12-490b79b6b6d1 None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:55:43.518747 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7554000-e152-42cb-ad12-490b79b6b6d1 None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:55:43.519047 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7554000-e152-42cb-ad12-490b79b6b6d1 None admin] PATH_INFO: `/v3/projects/4cad029b0df84051812bcec923a7adc6` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:55:43.520272 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7554000-e152-42cb-ad12-490b79b6b6d1 None admin] RBAC: Authorizing `identity:get_project(project_id=4cad029b0df84051812bcec923a7adc6)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:55:43.521715 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7554000-e152-42cb-ad12-490b79b6b6d1 None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:55:43.523325 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 567/1152] 10.222.0.50 () {56 vars in 1186 bytes} [Tue Sep 13 15:55:43 2022] GET /identity/v3/projects/4cad029b0df84051812bcec923a7adc6 => generated 311 bytes in 134 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:55:43.529103 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19faaf97-af3f-47c3-90a4-d75959b116e2 None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:55:43.625828 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19faaf97-af3f-47c3-90a4-d75959b116e2 None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:55:43.658880 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94752f8c-04be-4a9d-9335-888d9fbb60e3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:55:43.659548 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94752f8c-04be-4a9d-9335-888d9fbb60e3 None admin] REQUEST_METHOD: `PUT` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:55:43.659826 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94752f8c-04be-4a9d-9335-888d9fbb60e3 None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:55:43.660131 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94752f8c-04be-4a9d-9335-888d9fbb60e3 None admin] PATH_INFO: `/v3/projects/4cad029b0df84051812bcec923a7adc6/users/b2bf9837adc84a3c8a756f161413468f/roles/031d4e1a23e34f71858fc4a6dfbd4ccb` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:55:43.662112 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94752f8c-04be-4a9d-9335-888d9fbb60e3 None admin] RBAC: Authorizing `identity:create_grant(project_id=4cad029b0df84051812bcec923a7adc6, user_id=b2bf9837adc84a3c8a756f161413468f, role_id=031d4e1a23e34f71858fc4a6dfbd4ccb)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:55:43.663567 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94752f8c-04be-4a9d-9335-888d9fbb60e3 None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:55:43.676943 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 586/1153] 10.222.0.50 () {58 vars in 1439 bytes} [Tue Sep 13 15:55:43 2022] PUT /identity/v3/projects/4cad029b0df84051812bcec923a7adc6/users/b2bf9837adc84a3c8a756f161413468f/roles/031d4e1a23e34f71858fc4a6dfbd4ccb => generated 0 bytes in 148 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:55:45.795746 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef56d1ea-9133-4462-bc30-2970733f9a8d None None] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:55:45.796342 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef56d1ea-9133-4462-bc30-2970733f9a8d None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:55:45.796342 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef56d1ea-9133-4462-bc30-2970733f9a8d None None] PATH_INFO: `/` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:55:45.797620 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 568/1154] 10.222.0.50 () {54 vars in 927 bytes} [Tue Sep 13 15:55:45 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:55:45.802975 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0034890c-3a58-4833-8a84-0dd72dd0ca66 None None] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:55:45.803386 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0034890c-3a58-4833-8a84-0dd72dd0ca66 None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:55:45.803543 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0034890c-3a58-4833-8a84-0dd72dd0ca66 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:55:45.822158 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.auth.core [None req-0034890c-3a58-4833-8a84-0dd72dd0ca66 None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81437) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:55:45.823379 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-0034890c-3a58-4833-8a84-0dd72dd0ca66 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:55:45.849674 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:55:45.889114 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 587/1155] 10.222.0.50 () {58 vars in 1024 bytes} [Tue Sep 13 15:55:45 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 13 15:55:46.164338 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef56d1ea-9133-4462-bc30-2970733f9a8d None None] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:55:46.168168 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:55:46.265120 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef56d1ea-9133-4462-bc30-2970733f9a8d None None] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:55:46.297280 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fc06e3b-79a3-43c3-af88-2b27c4d82f18 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:55:46.297898 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fc06e3b-79a3-43c3-af88-2b27c4d82f18 None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:55:46.298178 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fc06e3b-79a3-43c3-af88-2b27c4d82f18 None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:55:46.298451 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fc06e3b-79a3-43c3-af88-2b27c4d82f18 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:55:46.309626 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fc06e3b-79a3-43c3-af88-2b27c4d82f18 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 13 15:55:46.314429 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application [None req-6fc06e3b-79a3-43c3-af88-2b27c4d82f18 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 13 15:55:46.314429 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:55:46.314429 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:55:46.314429 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:55:46.314429 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:55:46.314429 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:55:46.314429 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:55:46.314429 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:55:46.314429 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:55:46.314429 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:55:46.314429 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:55:46.314429 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:55:46.314429 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 13 15:55:46.314429 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 13 15:55:46.314429 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 13 15:55:46.314429 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 13 15:55:46.315799 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 13 15:55:46.315799 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 13 15:55:46.315799 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 13 15:55:46.315799 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 13 15:55:46.315799 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:55:46.315799 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:55:46.315799 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:55:46.315799 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:55:46.315799 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:55:46.315799 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:55:46.315799 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:55:46.315799 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application with Lock( Sep 13 15:55:46.315799 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:55:46.315799 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:55:46.315799 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:55:46.315799 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:55:46.317596 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:55:46.317596 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:55:46.317596 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:55:46.317596 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:55:46.317596 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 13 15:55:46.317596 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 13 15:55:46.317596 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 13 15:55:46.317596 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 13 15:55:46.317596 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 13 15:55:46.317596 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 13 15:55:46.317596 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 13 15:55:46.317596 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Sep 13 15:55:46.317596 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 569/1156] 10.222.0.50 () {56 vars in 1096 bytes} [Tue Sep 13 15:55:46 2022] GET /identity/v3/roles/admin => generated 83 bytes in 152 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:55:46.322163 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0034890c-3a58-4833-8a84-0dd72dd0ca66 None None] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:55:46.419921 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0034890c-3a58-4833-8a84-0dd72dd0ca66 None None] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:55:46.452645 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52fdc0ac-7f94-48bd-ae73-f431ec399f59 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:55:46.453233 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52fdc0ac-7f94-48bd-ae73-f431ec399f59 None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:55:46.453507 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52fdc0ac-7f94-48bd-ae73-f431ec399f59 None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:55:46.453772 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52fdc0ac-7f94-48bd-ae73-f431ec399f59 None admin] PATH_INFO: `/v3/roles` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:55:46.454414 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52fdc0ac-7f94-48bd-ae73-f431ec399f59 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81437) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:55:46.454710 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52fdc0ac-7f94-48bd-ae73-f431ec399f59 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:55:46.456099 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52fdc0ac-7f94-48bd-ae73-f431ec399f59 None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:55:46.463953 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 588/1157] 10.222.0.50 () {56 vars in 1099 bytes} [Tue Sep 13 15:55:46 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 142 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:55:46.469856 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fc06e3b-79a3-43c3-af88-2b27c4d82f18 None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:55:46.529911 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:55:46.567909 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fc06e3b-79a3-43c3-af88-2b27c4d82f18 None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:55:46.600204 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25e9ab12-a2f6-413f-9c3a-0c22c914a8a0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:55:46.600816 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25e9ab12-a2f6-413f-9c3a-0c22c914a8a0 None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:55:46.601092 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25e9ab12-a2f6-413f-9c3a-0c22c914a8a0 None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:55:46.601364 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25e9ab12-a2f6-413f-9c3a-0c22c914a8a0 None admin] PATH_INFO: `/v3/users/b2bf9837adc84a3c8a756f161413468f` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:55:46.602452 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25e9ab12-a2f6-413f-9c3a-0c22c914a8a0 None admin] RBAC: Authorizing `identity:get_user(user_id=b2bf9837adc84a3c8a756f161413468f)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:55:46.603888 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25e9ab12-a2f6-413f-9c3a-0c22c914a8a0 None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:55:46.605415 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 570/1158] 10.222.0.50 () {56 vars in 1177 bytes} [Tue Sep 13 15:55:46 2022] GET /identity/v3/users/b2bf9837adc84a3c8a756f161413468f => generated 286 bytes in 136 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:55:46.611132 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52fdc0ac-7f94-48bd-ae73-f431ec399f59 None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:55:46.638487 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:55:46.709345 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52fdc0ac-7f94-48bd-ae73-f431ec399f59 None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:55:46.741525 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-509944fb-a81e-467c-87fa-0ea570699733 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:55:46.742112 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-509944fb-a81e-467c-87fa-0ea570699733 None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:55:46.742415 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-509944fb-a81e-467c-87fa-0ea570699733 None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:55:46.742689 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-509944fb-a81e-467c-87fa-0ea570699733 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:55:46.807012 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-509944fb-a81e-467c-87fa-0ea570699733 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:55:46.808446 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-509944fb-a81e-467c-87fa-0ea570699733 None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:55:46.833521 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:55:46.903916 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 589/1159] 10.222.0.50 () {58 vars in 1282 bytes} [Tue Sep 13 15:55:46 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 293 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 13 15:55:46.909973 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25e9ab12-a2f6-413f-9c3a-0c22c914a8a0 None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:55:47.007996 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25e9ab12-a2f6-413f-9c3a-0c22c914a8a0 None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:55:47.036404 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:55:47.041468 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3e8f449-6f67-4f5c-a1ad-4489e4e9beee None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:55:47.042098 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3e8f449-6f67-4f5c-a1ad-4489e4e9beee None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:55:47.042378 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3e8f449-6f67-4f5c-a1ad-4489e4e9beee None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:55:47.042653 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3e8f449-6f67-4f5c-a1ad-4489e4e9beee None admin] PATH_INFO: `/v3/projects/4cad029b0df84051812bcec923a7adc6` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:55:47.043809 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3e8f449-6f67-4f5c-a1ad-4489e4e9beee None admin] RBAC: Authorizing `identity:get_project(project_id=4cad029b0df84051812bcec923a7adc6)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:55:47.045241 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3e8f449-6f67-4f5c-a1ad-4489e4e9beee None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:55:47.046873 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 571/1160] 10.222.0.50 () {56 vars in 1186 bytes} [Tue Sep 13 15:55:46 2022] GET /identity/v3/projects/4cad029b0df84051812bcec923a7adc6 => generated 311 bytes in 137 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:55:47.053498 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-509944fb-a81e-467c-87fa-0ea570699733 None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:55:47.149813 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-509944fb-a81e-467c-87fa-0ea570699733 None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:55:47.174614 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:55:47.186257 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c034a978-2ab1-47ee-b4b9-b8d00c701eb3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:55:47.186853 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c034a978-2ab1-47ee-b4b9-b8d00c701eb3 None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:55:47.187161 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c034a978-2ab1-47ee-b4b9-b8d00c701eb3 None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:55:47.187436 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c034a978-2ab1-47ee-b4b9-b8d00c701eb3 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:55:47.188148 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c034a978-2ab1-47ee-b4b9-b8d00c701eb3 None admin] RBAC: Adding query filter params (role.id=031d4e1a23e34f71858fc4a6dfbd4ccb, scope.project.id=4cad029b0df84051812bcec923a7adc6, user.id=b2bf9837adc84a3c8a756f161413468f) {{(pid=81437) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:55:47.188455 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c034a978-2ab1-47ee-b4b9-b8d00c701eb3 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:55:47.189882 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c034a978-2ab1-47ee-b4b9-b8d00c701eb3 None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:55:47.195712 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 590/1161] 10.222.0.50 () {56 vars in 1374 bytes} [Tue Sep 13 15:55:47 2022] GET /identity/v3/role_assignments?user.id=b2bf9837adc84a3c8a756f161413468f&scope.project.id=4cad029b0df84051812bcec923a7adc6&role.id=031d4e1a23e34f71858fc4a6dfbd4ccb => generated 608 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:55:49.366063 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3026080-99f4-4d0d-a5f2-3882674b56e4 None None] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:55:49.366593 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3026080-99f4-4d0d-a5f2-3882674b56e4 None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:55:49.366593 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3026080-99f4-4d0d-a5f2-3882674b56e4 None None] PATH_INFO: `/` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:55:49.367956 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 572/1162] 10.222.0.50 () {54 vars in 927 bytes} [Tue Sep 13 15:55:49 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:55:49.373062 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a59d8d97-68e1-420d-8ae6-4298c77edcde None None] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:55:49.373484 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a59d8d97-68e1-420d-8ae6-4298c77edcde None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:55:49.373613 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a59d8d97-68e1-420d-8ae6-4298c77edcde None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:55:49.391396 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.auth.core [None req-a59d8d97-68e1-420d-8ae6-4298c77edcde None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81437) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:55:49.392570 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-a59d8d97-68e1-420d-8ae6-4298c77edcde None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:55:49.457149 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 591/1163] 10.222.0.50 () {58 vars in 1024 bytes} [Tue Sep 13 15:55:49 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 13 15:55:49.725981 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3026080-99f4-4d0d-a5f2-3882674b56e4 None None] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:55:49.825705 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3026080-99f4-4d0d-a5f2-3882674b56e4 None None] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:55:49.858969 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ff2f84b-fd25-47fa-9254-af75c48b55ac None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:55:49.859562 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ff2f84b-fd25-47fa-9254-af75c48b55ac None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:55:49.859842 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ff2f84b-fd25-47fa-9254-af75c48b55ac None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:55:49.860118 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ff2f84b-fd25-47fa-9254-af75c48b55ac None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:55:49.923171 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:55:49.925558 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ff2f84b-fd25-47fa-9254-af75c48b55ac None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:55:49.927470 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ff2f84b-fd25-47fa-9254-af75c48b55ac None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:55:50.022047 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 573/1164] 10.222.0.50 () {58 vars in 1282 bytes} [Tue Sep 13 15:55:49 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 297 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 13 15:55:50.027945 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a59d8d97-68e1-420d-8ae6-4298c77edcde None None] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:55:50.125559 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a59d8d97-68e1-420d-8ae6-4298c77edcde None None] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:55:50.145738 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:55:50.164163 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5eb26f7a-846e-4f4b-afcd-a0e0d8484f96 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:55:50.164789 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5eb26f7a-846e-4f4b-afcd-a0e0d8484f96 None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:55:50.165064 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5eb26f7a-846e-4f4b-afcd-a0e0d8484f96 None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:55:50.165334 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5eb26f7a-846e-4f4b-afcd-a0e0d8484f96 None admin] PATH_INFO: `/v3/domains/swift_test` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:55:50.171920 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5eb26f7a-846e-4f4b-afcd-a0e0d8484f96 None admin] RBAC: Authorizing `identity:get_domain(domain_id=swift_test)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:55:50.173347 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5eb26f7a-846e-4f4b-afcd-a0e0d8484f96 None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:55:50.185293 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application [None req-5eb26f7a-846e-4f4b-afcd-a0e0d8484f96 None admin] Could not find domain: swift_test.: keystone.exception.DomainNotFound: Could not find domain: swift_test. Sep 13 15:55:50.185293 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:55:50.185293 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 712, in get_domain Sep 13 15:55:50.185293 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application project = self.driver.get_project(domain_id) Sep 13 15:55:50.185293 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 13 15:55:50.185293 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 13 15:55:50.185293 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 13 15:55:50.185293 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 13 15:55:50.185293 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: swift_test. Sep 13 15:55:50.185293 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application Sep 13 15:55:50.185293 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 13 15:55:50.185293 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application Sep 13 15:55:50.185293 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:55:50.185293 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:55:50.185293 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:55:50.185293 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:55:50.185293 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:55:50.186528 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:55:50.186528 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:55:50.186528 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:55:50.186528 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:55:50.186528 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:55:50.186528 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:55:50.186528 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 13 15:55:50.186528 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 13 15:55:50.186528 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 13 15:55:50.186528 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 13 15:55:50.186528 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:55:50.186528 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:55:50.186528 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:55:50.186528 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:55:50.186528 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:55:50.186528 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:55:50.187636 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:55:50.187636 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application with Lock( Sep 13 15:55:50.187636 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:55:50.187636 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:55:50.187636 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:55:50.187636 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:55:50.187636 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:55:50.187636 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:55:50.187636 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:55:50.187636 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:55:50.187636 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 718, in get_domain Sep 13 15:55:50.187636 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 13 15:55:50.187636 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: swift_test. Sep 13 15:55:50.187636 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application Sep 13 15:55:50.187636 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 592/1165] 10.222.0.50 () {56 vars in 1117 bytes} [Tue Sep 13 15:55:50 2022] GET /identity/v3/domains/swift_test => generated 90 bytes in 159 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:55:50.192577 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ff2f84b-fd25-47fa-9254-af75c48b55ac None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:55:50.286092 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:55:50.289180 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ff2f84b-fd25-47fa-9254-af75c48b55ac None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:55:50.322876 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d38eaa5c-a4f0-4c42-96db-bc91d5cd4243 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:55:50.323495 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d38eaa5c-a4f0-4c42-96db-bc91d5cd4243 None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:55:50.323770 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d38eaa5c-a4f0-4c42-96db-bc91d5cd4243 None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:55:50.324038 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d38eaa5c-a4f0-4c42-96db-bc91d5cd4243 None admin] PATH_INFO: `/v3/domains` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:55:50.324663 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d38eaa5c-a4f0-4c42-96db-bc91d5cd4243 None admin] RBAC: Adding query filter params (name=swift_test) {{(pid=81438) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:55:50.324959 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d38eaa5c-a4f0-4c42-96db-bc91d5cd4243 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:55:50.326299 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d38eaa5c-a4f0-4c42-96db-bc91d5cd4243 None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:55:50.333156 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 574/1166] 10.222.0.50 () {56 vars in 1115 bytes} [Tue Sep 13 15:55:50 2022] GET /identity/v3/domains?name=swift_test => generated 125 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:55:52.481016 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b97e91f5-c2d5-4736-bfe5-ae8bc79c9185 None None] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:55:52.481582 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b97e91f5-c2d5-4736-bfe5-ae8bc79c9185 None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:55:52.481582 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b97e91f5-c2d5-4736-bfe5-ae8bc79c9185 None None] PATH_INFO: `/` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:55:52.482875 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 593/1167] 10.222.0.50 () {54 vars in 927 bytes} [Tue Sep 13 15:55:52 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:55:52.488265 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-192b38ff-0d44-4d21-a6a8-73f483b2ff40 None None] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:55:52.488766 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-192b38ff-0d44-4d21-a6a8-73f483b2ff40 None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:55:52.488912 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-192b38ff-0d44-4d21-a6a8-73f483b2ff40 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:55:52.507133 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.auth.core [None req-192b38ff-0d44-4d21-a6a8-73f483b2ff40 None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81438) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:55:52.508307 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-192b38ff-0d44-4d21-a6a8-73f483b2ff40 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:55:52.572545 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 575/1168] 10.222.0.50 () {58 vars in 1024 bytes} [Tue Sep 13 15:55:52 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 13 15:55:52.843909 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b97e91f5-c2d5-4736-bfe5-ae8bc79c9185 None None] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:55:52.950237 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b97e91f5-c2d5-4736-bfe5-ae8bc79c9185 None None] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:55:52.982706 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2100ef80-abdf-497d-973a-e22ec804340b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:55:52.983324 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2100ef80-abdf-497d-973a-e22ec804340b None admin] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:55:52.983603 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2100ef80-abdf-497d-973a-e22ec804340b None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:55:52.983877 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2100ef80-abdf-497d-973a-e22ec804340b None admin] PATH_INFO: `/v3/domains` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:55:52.984543 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2100ef80-abdf-497d-973a-e22ec804340b None admin] RBAC: Authorizing `identity:create_domain()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:55:52.985910 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2100ef80-abdf-497d-973a-e22ec804340b None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:55:53.002143 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 594/1169] 10.222.0.50 () {60 vars in 1138 bytes} [Tue Sep 13 15:55:52 2022] POST /identity/v3/domains => generated 267 bytes in 159 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 15:55:55.170298 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcc7a1e4-0593-41bb-951f-0bd9bf70d4be None None] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:55:55.170845 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcc7a1e4-0593-41bb-951f-0bd9bf70d4be None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:55:55.170845 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcc7a1e4-0593-41bb-951f-0bd9bf70d4be None None] PATH_INFO: `/` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:55:55.172217 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 576/1170] 10.222.0.50 () {54 vars in 927 bytes} [Tue Sep 13 15:55:55 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:55:55.177600 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83de77a3-df37-43cf-bf8e-c6d8f5d7f14c None None] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:55:55.178041 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83de77a3-df37-43cf-bf8e-c6d8f5d7f14c None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:55:55.178181 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83de77a3-df37-43cf-bf8e-c6d8f5d7f14c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:55:55.196282 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.auth.core [None req-83de77a3-df37-43cf-bf8e-c6d8f5d7f14c None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81437) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:55:55.197451 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-83de77a3-df37-43cf-bf8e-c6d8f5d7f14c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:55:55.231257 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:55:55.264518 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 595/1171] 10.222.0.50 () {58 vars in 1024 bytes} [Tue Sep 13 15:55:55 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 13 15:55:55.534817 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcc7a1e4-0593-41bb-951f-0bd9bf70d4be None None] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:55:55.625965 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:55:55.634026 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcc7a1e4-0593-41bb-951f-0bd9bf70d4be None None] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:55:55.666830 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2987fc9-6160-4ebe-a070-c2d54d7d19c2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:55:55.667475 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2987fc9-6160-4ebe-a070-c2d54d7d19c2 None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:55:55.667759 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2987fc9-6160-4ebe-a070-c2d54d7d19c2 None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:55:55.668035 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2987fc9-6160-4ebe-a070-c2d54d7d19c2 None admin] PATH_INFO: `/v3/domains/621e52af32904b958a8daeddf1e41592` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:55:55.678804 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2987fc9-6160-4ebe-a070-c2d54d7d19c2 None admin] RBAC: Authorizing `identity:get_domain(domain_id=621e52af32904b958a8daeddf1e41592)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:55:55.680518 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2987fc9-6160-4ebe-a070-c2d54d7d19c2 None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:55:55.682079 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 577/1172] 10.222.0.50 () {56 vars in 1183 bytes} [Tue Sep 13 15:55:55 2022] GET /identity/v3/domains/621e52af32904b958a8daeddf1e41592 => generated 267 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:55:55.687837 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83de77a3-df37-43cf-bf8e-c6d8f5d7f14c None None] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:55:55.784296 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83de77a3-df37-43cf-bf8e-c6d8f5d7f14c None None] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:55:55.818129 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7e30d00-0106-4afb-8818-0a3dd3537a3b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:55:55.818673 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7e30d00-0106-4afb-8818-0a3dd3537a3b None admin] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:55:55.818983 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7e30d00-0106-4afb-8818-0a3dd3537a3b None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:55:55.819260 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7e30d00-0106-4afb-8818-0a3dd3537a3b None admin] PATH_INFO: `/v3/projects` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:55:55.819899 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7e30d00-0106-4afb-8818-0a3dd3537a3b None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:55:55.821321 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7e30d00-0106-4afb-8818-0a3dd3537a3b None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:55:55.846857 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 596/1173] 10.222.0.50 () {60 vars in 1141 bytes} [Tue Sep 13 15:55:55 2022] POST /identity/v3/projects => generated 361 bytes in 159 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 15:55:58.006206 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00ddbf26-8910-486f-b5df-677fbf11d627 None None] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:55:58.006772 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00ddbf26-8910-486f-b5df-677fbf11d627 None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:55:58.006772 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00ddbf26-8910-486f-b5df-677fbf11d627 None None] PATH_INFO: `/` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:55:58.008137 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 578/1174] 10.222.0.50 () {54 vars in 927 bytes} [Tue Sep 13 15:55:58 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:55:58.013428 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e075338a-9a15-450d-bda1-be112941e290 None None] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:55:58.013838 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e075338a-9a15-450d-bda1-be112941e290 None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:55:58.013980 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e075338a-9a15-450d-bda1-be112941e290 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:55:58.031991 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.auth.core [None req-e075338a-9a15-450d-bda1-be112941e290 None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81437) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:55:58.033173 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-e075338a-9a15-450d-bda1-be112941e290 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:55:58.049643 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:55:58.101060 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 597/1175] 10.222.0.50 () {58 vars in 1024 bytes} [Tue Sep 13 15:55:58 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 13 15:55:58.372183 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00ddbf26-8910-486f-b5df-677fbf11d627 None None] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:55:58.474518 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00ddbf26-8910-486f-b5df-677fbf11d627 None None] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:55:58.511734 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ac68467-3eb2-4d93-908f-59767f646c63 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:55:58.512347 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ac68467-3eb2-4d93-908f-59767f646c63 None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:55:58.512623 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ac68467-3eb2-4d93-908f-59767f646c63 None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:55:58.512888 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ac68467-3eb2-4d93-908f-59767f646c63 None admin] PATH_INFO: `/v3/domains/621e52af32904b958a8daeddf1e41592` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:55:58.513979 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ac68467-3eb2-4d93-908f-59767f646c63 None admin] RBAC: Authorizing `identity:get_domain(domain_id=621e52af32904b958a8daeddf1e41592)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:55:58.515426 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ac68467-3eb2-4d93-908f-59767f646c63 None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:55:58.516951 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 579/1176] 10.222.0.50 () {56 vars in 1183 bytes} [Tue Sep 13 15:55:58 2022] GET /identity/v3/domains/621e52af32904b958a8daeddf1e41592 => generated 267 bytes in 145 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:55:58.523392 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e075338a-9a15-450d-bda1-be112941e290 None None] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:55:58.618677 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e075338a-9a15-450d-bda1-be112941e290 None None] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:55:58.652804 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2032294a-71cf-4275-860c-bdc6703a5cb3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:55:58.653332 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2032294a-71cf-4275-860c-bdc6703a5cb3 None admin] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:55:58.653610 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2032294a-71cf-4275-860c-bdc6703a5cb3 None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:55:58.653882 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2032294a-71cf-4275-860c-bdc6703a5cb3 None admin] PATH_INFO: `/v3/users` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:55:58.654528 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2032294a-71cf-4275-860c-bdc6703a5cb3 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:55:58.655984 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2032294a-71cf-4275-860c-bdc6703a5cb3 None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:55:58.677678 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 598/1177] 10.222.0.50 () {60 vars in 1132 bytes} [Tue Sep 13 15:55:58 2022] POST /identity/v3/users => generated 311 bytes in 155 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 15:56:00.861093 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc6b5f54-8520-4b31-ab97-fd826960ffd1 None None] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:56:00.861658 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc6b5f54-8520-4b31-ab97-fd826960ffd1 None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:56:00.861658 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc6b5f54-8520-4b31-ab97-fd826960ffd1 None None] PATH_INFO: `/` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:56:00.862981 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 580/1178] 10.222.0.50 () {54 vars in 927 bytes} [Tue Sep 13 15:56:00 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:56:00.868546 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2887538e-b5b3-4d1d-a9eb-e069841c2399 None None] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:56:00.868981 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2887538e-b5b3-4d1d-a9eb-e069841c2399 None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:56:00.869083 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2887538e-b5b3-4d1d-a9eb-e069841c2399 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:56:00.887546 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.auth.core [None req-2887538e-b5b3-4d1d-a9eb-e069841c2399 None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81437) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:56:00.888709 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-2887538e-b5b3-4d1d-a9eb-e069841c2399 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:56:00.890613 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:56:00.956553 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 599/1179] 10.222.0.50 () {58 vars in 1024 bytes} [Tue Sep 13 15:56:00 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 13 15:56:01.232859 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc6b5f54-8520-4b31-ab97-fd826960ffd1 None None] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:56:01.258256 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:56:01.335597 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc6b5f54-8520-4b31-ab97-fd826960ffd1 None None] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:56:01.368273 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d218f0c-0014-4d1c-9dc5-d6c009822cd6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:56:01.368892 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d218f0c-0014-4d1c-9dc5-d6c009822cd6 None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:56:01.369172 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d218f0c-0014-4d1c-9dc5-d6c009822cd6 None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:56:01.369446 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d218f0c-0014-4d1c-9dc5-d6c009822cd6 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:56:01.380362 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d218f0c-0014-4d1c-9dc5-d6c009822cd6 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 13 15:56:01.385039 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application [None req-8d218f0c-0014-4d1c-9dc5-d6c009822cd6 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 13 15:56:01.385039 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:56:01.385039 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:56:01.385039 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:56:01.385039 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:56:01.385039 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:56:01.385039 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:56:01.385039 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:56:01.385039 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:56:01.385039 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:56:01.385039 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:56:01.385039 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:56:01.385039 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 13 15:56:01.385039 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 13 15:56:01.385039 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 13 15:56:01.385039 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 13 15:56:01.386121 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 13 15:56:01.386121 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 13 15:56:01.386121 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 13 15:56:01.386121 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 13 15:56:01.386121 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:56:01.386121 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:56:01.386121 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:56:01.386121 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:56:01.386121 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:56:01.386121 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:56:01.386121 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:56:01.386121 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application with Lock( Sep 13 15:56:01.386121 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:56:01.386121 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:56:01.386121 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:56:01.386121 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:56:01.387187 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:56:01.387187 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:56:01.387187 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:56:01.387187 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:56:01.387187 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 13 15:56:01.387187 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 13 15:56:01.387187 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 13 15:56:01.387187 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 13 15:56:01.387187 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 13 15:56:01.387187 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 13 15:56:01.387187 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 13 15:56:01.387187 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Sep 13 15:56:01.387187 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 581/1180] 10.222.0.50 () {56 vars in 1096 bytes} [Tue Sep 13 15:56:01 2022] GET /identity/v3/roles/admin => generated 83 bytes in 154 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:56:01.392132 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2887538e-b5b3-4d1d-a9eb-e069841c2399 None None] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:56:01.486967 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2887538e-b5b3-4d1d-a9eb-e069841c2399 None None] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:56:01.519107 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2a245de-4e90-477e-896d-8a438dbedbdc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:56:01.520389 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2a245de-4e90-477e-896d-8a438dbedbdc None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:56:01.520667 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2a245de-4e90-477e-896d-8a438dbedbdc None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:56:01.520936 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2a245de-4e90-477e-896d-8a438dbedbdc None admin] PATH_INFO: `/v3/roles` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:56:01.521585 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2a245de-4e90-477e-896d-8a438dbedbdc None admin] RBAC: Adding query filter params (name=admin) {{(pid=81437) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:56:01.521888 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2a245de-4e90-477e-896d-8a438dbedbdc None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:56:01.523258 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2a245de-4e90-477e-896d-8a438dbedbdc None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:56:01.528028 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:56:01.531847 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 600/1181] 10.222.0.50 () {56 vars in 1099 bytes} [Tue Sep 13 15:56:01 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:56:01.537504 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d218f0c-0014-4d1c-9dc5-d6c009822cd6 None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:56:01.617061 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:56:01.635692 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d218f0c-0014-4d1c-9dc5-d6c009822cd6 None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:56:01.667896 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23bae358-f080-43b1-b045-eb22751d0023 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:56:01.668512 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23bae358-f080-43b1-b045-eb22751d0023 None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:56:01.668793 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23bae358-f080-43b1-b045-eb22751d0023 None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:56:01.669069 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23bae358-f080-43b1-b045-eb22751d0023 None admin] PATH_INFO: `/v3/users/9596c91ee75e4abbbe5ac04486da0981` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:56:01.682709 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23bae358-f080-43b1-b045-eb22751d0023 None admin] RBAC: Authorizing `identity:get_user(user_id=9596c91ee75e4abbbe5ac04486da0981)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:56:01.684168 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23bae358-f080-43b1-b045-eb22751d0023 None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:56:01.685722 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 582/1182] 10.222.0.50 () {56 vars in 1177 bytes} [Tue Sep 13 15:56:01 2022] GET /identity/v3/users/9596c91ee75e4abbbe5ac04486da0981 => generated 311 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:56:01.692142 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2a245de-4e90-477e-896d-8a438dbedbdc None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:56:01.787231 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2a245de-4e90-477e-896d-8a438dbedbdc None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:56:01.819832 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69293eb3-6540-486f-98e7-c750c1d156b0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:56:01.820430 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69293eb3-6540-486f-98e7-c750c1d156b0 None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:56:01.820711 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69293eb3-6540-486f-98e7-c750c1d156b0 None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:56:01.820983 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69293eb3-6540-486f-98e7-c750c1d156b0 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:56:01.884175 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:56:01.886544 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69293eb3-6540-486f-98e7-c750c1d156b0 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:56:01.888439 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69293eb3-6540-486f-98e7-c750c1d156b0 None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:56:01.984125 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 601/1183] 10.222.0.50 () {58 vars in 1282 bytes} [Tue Sep 13 15:56:01 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 293 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 13 15:56:01.990737 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23bae358-f080-43b1-b045-eb22751d0023 None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:56:02.089081 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23bae358-f080-43b1-b045-eb22751d0023 None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:56:02.121484 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea8057f0-cac3-4adb-b2d4-71a88938b290 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:56:02.122103 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea8057f0-cac3-4adb-b2d4-71a88938b290 None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:56:02.122381 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea8057f0-cac3-4adb-b2d4-71a88938b290 None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:56:02.122655 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea8057f0-cac3-4adb-b2d4-71a88938b290 None admin] PATH_INFO: `/v3/projects/642b5b41c00f4463ac985538cd5ef6b9` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:56:02.123850 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea8057f0-cac3-4adb-b2d4-71a88938b290 None admin] RBAC: Authorizing `identity:get_project(project_id=642b5b41c00f4463ac985538cd5ef6b9)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:56:02.125272 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea8057f0-cac3-4adb-b2d4-71a88938b290 None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:56:02.126893 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 583/1184] 10.222.0.50 () {56 vars in 1186 bytes} [Tue Sep 13 15:56:01 2022] GET /identity/v3/projects/642b5b41c00f4463ac985538cd5ef6b9 => generated 361 bytes in 137 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:56:02.133534 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69293eb3-6540-486f-98e7-c750c1d156b0 None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:56:02.226645 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:56:02.229733 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69293eb3-6540-486f-98e7-c750c1d156b0 None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:56:02.263225 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae8f79d4-77dc-4863-9837-41d29fd325ea None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:56:02.263822 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae8f79d4-77dc-4863-9837-41d29fd325ea None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:56:02.264099 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae8f79d4-77dc-4863-9837-41d29fd325ea None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:56:02.264369 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae8f79d4-77dc-4863-9837-41d29fd325ea None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:56:02.265079 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae8f79d4-77dc-4863-9837-41d29fd325ea None admin] RBAC: Adding query filter params (role.id=031d4e1a23e34f71858fc4a6dfbd4ccb, scope.project.id=642b5b41c00f4463ac985538cd5ef6b9, user.id=9596c91ee75e4abbbe5ac04486da0981) {{(pid=81437) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:56:02.265385 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae8f79d4-77dc-4863-9837-41d29fd325ea None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:56:02.266831 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae8f79d4-77dc-4863-9837-41d29fd325ea None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:56:02.272720 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 602/1185] 10.222.0.50 () {56 vars in 1374 bytes} [Tue Sep 13 15:56:02 2022] GET /identity/v3/role_assignments?user.id=9596c91ee75e4abbbe5ac04486da0981&scope.project.id=642b5b41c00f4463ac985538cd5ef6b9&role.id=031d4e1a23e34f71858fc4a6dfbd4ccb => generated 259 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:56:04.402498 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35f856d1-8028-4170-85e7-fd5ceb52f0a1 None None] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:56:04.403076 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35f856d1-8028-4170-85e7-fd5ceb52f0a1 None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:56:04.403076 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35f856d1-8028-4170-85e7-fd5ceb52f0a1 None None] PATH_INFO: `/` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:56:04.404390 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 584/1186] 10.222.0.50 () {54 vars in 927 bytes} [Tue Sep 13 15:56:04 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:56:04.409896 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f562a01-19c6-4e63-89fe-9679c9bf8f40 None None] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:56:04.410340 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f562a01-19c6-4e63-89fe-9679c9bf8f40 None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:56:04.410627 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f562a01-19c6-4e63-89fe-9679c9bf8f40 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:56:04.429240 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.auth.core [None req-2f562a01-19c6-4e63-89fe-9679c9bf8f40 None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81437) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:56:04.430688 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-2f562a01-19c6-4e63-89fe-9679c9bf8f40 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:56:04.496135 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 603/1187] 10.222.0.50 () {58 vars in 1024 bytes} [Tue Sep 13 15:56:04 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 13 15:56:04.768431 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35f856d1-8028-4170-85e7-fd5ceb52f0a1 None None] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:56:04.779496 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:56:04.868402 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35f856d1-8028-4170-85e7-fd5ceb52f0a1 None None] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:56:04.901094 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62940040-af68-4b35-874f-614693c6d709 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:56:04.901706 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62940040-af68-4b35-874f-614693c6d709 None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:56:04.901985 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62940040-af68-4b35-874f-614693c6d709 None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:56:04.902261 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62940040-af68-4b35-874f-614693c6d709 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:56:04.912023 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62940040-af68-4b35-874f-614693c6d709 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 13 15:56:04.916778 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application [None req-62940040-af68-4b35-874f-614693c6d709 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 13 15:56:04.916778 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:56:04.916778 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:56:04.916778 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:56:04.916778 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:56:04.916778 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:56:04.916778 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:56:04.916778 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:56:04.916778 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:56:04.916778 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:56:04.916778 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:56:04.916778 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:56:04.916778 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 13 15:56:04.916778 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 13 15:56:04.916778 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 13 15:56:04.916778 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 13 15:56:04.917801 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 13 15:56:04.917801 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 13 15:56:04.917801 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 13 15:56:04.917801 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 13 15:56:04.917801 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:56:04.917801 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:56:04.917801 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:56:04.917801 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:56:04.917801 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:56:04.917801 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:56:04.917801 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:56:04.917801 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application with Lock( Sep 13 15:56:04.917801 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:56:04.917801 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:56:04.917801 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:56:04.917801 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:56:04.918750 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:56:04.918750 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:56:04.918750 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:56:04.918750 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:56:04.918750 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 13 15:56:04.918750 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 13 15:56:04.918750 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 13 15:56:04.918750 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 13 15:56:04.918750 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 13 15:56:04.918750 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 13 15:56:04.918750 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 13 15:56:04.918750 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Sep 13 15:56:04.918750 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 585/1188] 10.222.0.50 () {56 vars in 1096 bytes} [Tue Sep 13 15:56:04 2022] GET /identity/v3/roles/admin => generated 83 bytes in 150 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:56:04.923956 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f562a01-19c6-4e63-89fe-9679c9bf8f40 None None] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:56:05.013938 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:56:05.020705 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f562a01-19c6-4e63-89fe-9679c9bf8f40 None None] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:56:05.053989 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a72403a-c75e-404c-9976-eafe5467a5c1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:56:05.054593 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a72403a-c75e-404c-9976-eafe5467a5c1 None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:56:05.054872 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a72403a-c75e-404c-9976-eafe5467a5c1 None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:56:05.055423 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a72403a-c75e-404c-9976-eafe5467a5c1 None admin] PATH_INFO: `/v3/roles` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:56:05.056078 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a72403a-c75e-404c-9976-eafe5467a5c1 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81437) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:56:05.056377 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a72403a-c75e-404c-9976-eafe5467a5c1 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:56:05.057744 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a72403a-c75e-404c-9976-eafe5467a5c1 None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:56:05.065680 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 604/1189] 10.222.0.50 () {56 vars in 1099 bytes} [Tue Sep 13 15:56:04 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 142 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 13 15:56:05.072206 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62940040-af68-4b35-874f-614693c6d709 None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:56:05.137047 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:56:05.169792 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62940040-af68-4b35-874f-614693c6d709 None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:56:05.203277 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-912a9560-0a8b-48a7-895e-5578d17f2060 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:56:05.203891 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-912a9560-0a8b-48a7-895e-5578d17f2060 None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:56:05.204170 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-912a9560-0a8b-48a7-895e-5578d17f2060 None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:56:05.204445 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-912a9560-0a8b-48a7-895e-5578d17f2060 None admin] PATH_INFO: `/v3/users/9596c91ee75e4abbbe5ac04486da0981` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:56:05.205547 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-912a9560-0a8b-48a7-895e-5578d17f2060 None admin] RBAC: Authorizing `identity:get_user(user_id=9596c91ee75e4abbbe5ac04486da0981)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:56:05.207081 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-912a9560-0a8b-48a7-895e-5578d17f2060 None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:56:05.208629 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 586/1190] 10.222.0.50 () {56 vars in 1177 bytes} [Tue Sep 13 15:56:05 2022] GET /identity/v3/users/9596c91ee75e4abbbe5ac04486da0981 => generated 311 bytes in 137 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:56:05.214414 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a72403a-c75e-404c-9976-eafe5467a5c1 None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:56:05.312088 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a72403a-c75e-404c-9976-eafe5467a5c1 None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:56:05.344325 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ae525bc-a8df-4e3f-9c52-eff1324f66e5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:56:05.344939 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ae525bc-a8df-4e3f-9c52-eff1324f66e5 None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:56:05.345216 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ae525bc-a8df-4e3f-9c52-eff1324f66e5 None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:56:05.345487 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ae525bc-a8df-4e3f-9c52-eff1324f66e5 None admin] PATH_INFO: `/v3/projects/642b5b41c00f4463ac985538cd5ef6b9` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:56:05.346561 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ae525bc-a8df-4e3f-9c52-eff1324f66e5 None admin] RBAC: Authorizing `identity:get_project(project_id=642b5b41c00f4463ac985538cd5ef6b9)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:56:05.348513 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ae525bc-a8df-4e3f-9c52-eff1324f66e5 None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:56:05.350102 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 605/1191] 10.222.0.50 () {56 vars in 1186 bytes} [Tue Sep 13 15:56:05 2022] GET /identity/v3/projects/642b5b41c00f4463ac985538cd5ef6b9 => generated 361 bytes in 136 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:56:05.356021 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-912a9560-0a8b-48a7-895e-5578d17f2060 None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:56:05.452649 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-912a9560-0a8b-48a7-895e-5578d17f2060 None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:56:05.485857 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-261f0d9c-792b-4938-bbf7-aac50244677b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:56:05.486497 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-261f0d9c-792b-4938-bbf7-aac50244677b None admin] REQUEST_METHOD: `PUT` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:56:05.486777 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-261f0d9c-792b-4938-bbf7-aac50244677b None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:56:05.487080 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-261f0d9c-792b-4938-bbf7-aac50244677b None admin] PATH_INFO: `/v3/projects/642b5b41c00f4463ac985538cd5ef6b9/users/9596c91ee75e4abbbe5ac04486da0981/roles/031d4e1a23e34f71858fc4a6dfbd4ccb` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:56:05.489107 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-261f0d9c-792b-4938-bbf7-aac50244677b None admin] RBAC: Authorizing `identity:create_grant(project_id=642b5b41c00f4463ac985538cd5ef6b9, user_id=9596c91ee75e4abbbe5ac04486da0981, role_id=031d4e1a23e34f71858fc4a6dfbd4ccb)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:56:05.490510 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-261f0d9c-792b-4938-bbf7-aac50244677b None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:56:05.493455 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:56:05.503711 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 587/1192] 10.222.0.50 () {58 vars in 1439 bytes} [Tue Sep 13 15:56:05 2022] PUT /identity/v3/projects/642b5b41c00f4463ac985538cd5ef6b9/users/9596c91ee75e4abbbe5ac04486da0981/roles/031d4e1a23e34f71858fc4a6dfbd4ccb => generated 0 bytes in 148 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:56:07.614553 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e43fad3f-352d-4d4e-b92a-f07c3af34963 None None] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:56:07.615136 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e43fad3f-352d-4d4e-b92a-f07c3af34963 None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:56:07.615136 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e43fad3f-352d-4d4e-b92a-f07c3af34963 None None] PATH_INFO: `/` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:56:07.616456 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 606/1193] 10.222.0.50 () {54 vars in 927 bytes} [Tue Sep 13 15:56:07 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:56:07.621627 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db386a04-6c77-4b48-a6ff-26674aa4198b None None] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:56:07.621997 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db386a04-6c77-4b48-a6ff-26674aa4198b None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:56:07.622184 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db386a04-6c77-4b48-a6ff-26674aa4198b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:56:07.640488 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.auth.core [None req-db386a04-6c77-4b48-a6ff-26674aa4198b None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81438) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:56:07.641658 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-db386a04-6c77-4b48-a6ff-26674aa4198b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:56:07.706385 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 588/1194] 10.222.0.50 () {58 vars in 1024 bytes} [Tue Sep 13 15:56:07 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 13 15:56:07.974658 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e43fad3f-352d-4d4e-b92a-f07c3af34963 None None] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:56:07.999985 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:56:08.075877 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e43fad3f-352d-4d4e-b92a-f07c3af34963 None None] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:56:08.108708 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc73c21e-be3e-456b-af6a-f47dbabea0b9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:56:08.109334 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc73c21e-be3e-456b-af6a-f47dbabea0b9 None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:56:08.109617 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc73c21e-be3e-456b-af6a-f47dbabea0b9 None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:56:08.109894 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc73c21e-be3e-456b-af6a-f47dbabea0b9 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:56:08.120379 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc73c21e-be3e-456b-af6a-f47dbabea0b9 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 13 15:56:08.125140 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application [None req-dc73c21e-be3e-456b-af6a-f47dbabea0b9 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 13 15:56:08.125140 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:56:08.125140 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:56:08.125140 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:56:08.125140 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:56:08.125140 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:56:08.125140 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:56:08.125140 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:56:08.125140 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:56:08.125140 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:56:08.125140 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:56:08.125140 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:56:08.125140 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 13 15:56:08.125140 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 13 15:56:08.125140 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 13 15:56:08.125140 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 13 15:56:08.126288 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 13 15:56:08.126288 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 13 15:56:08.126288 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 13 15:56:08.126288 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 13 15:56:08.126288 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:56:08.126288 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:56:08.126288 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:56:08.126288 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:56:08.126288 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:56:08.126288 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:56:08.126288 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:56:08.126288 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application with Lock( Sep 13 15:56:08.126288 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:56:08.126288 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:56:08.126288 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:56:08.126288 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:56:08.127212 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:56:08.127212 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:56:08.127212 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:56:08.127212 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:56:08.127212 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 13 15:56:08.127212 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 13 15:56:08.127212 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 13 15:56:08.127212 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 13 15:56:08.127212 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 13 15:56:08.127212 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 13 15:56:08.127212 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 13 15:56:08.127212 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application Sep 13 15:56:08.127212 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 607/1195] 10.222.0.50 () {56 vars in 1096 bytes} [Tue Sep 13 15:56:07 2022] GET /identity/v3/roles/admin => generated 83 bytes in 152 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:56:08.132318 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db386a04-6c77-4b48-a6ff-26674aa4198b None None] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:56:08.242707 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db386a04-6c77-4b48-a6ff-26674aa4198b None None] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:56:08.266564 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:56:08.275573 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d55d8e17-4bd4-4d5d-90bf-b8e8caf89ed7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:56:08.276174 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d55d8e17-4bd4-4d5d-90bf-b8e8caf89ed7 None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:56:08.276458 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d55d8e17-4bd4-4d5d-90bf-b8e8caf89ed7 None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:56:08.276734 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d55d8e17-4bd4-4d5d-90bf-b8e8caf89ed7 None admin] PATH_INFO: `/v3/roles` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:56:08.277393 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d55d8e17-4bd4-4d5d-90bf-b8e8caf89ed7 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81438) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:56:08.277696 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d55d8e17-4bd4-4d5d-90bf-b8e8caf89ed7 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:56:08.279090 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d55d8e17-4bd4-4d5d-90bf-b8e8caf89ed7 None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:56:08.287612 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 589/1196] 10.222.0.50 () {56 vars in 1099 bytes} [Tue Sep 13 15:56:08 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:56:08.293437 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc73c21e-be3e-456b-af6a-f47dbabea0b9 None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:56:08.375761 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:56:08.391550 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc73c21e-be3e-456b-af6a-f47dbabea0b9 None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:56:08.427065 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf50a238-44b5-48e9-87ca-8bdf943aa203 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:56:08.427685 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf50a238-44b5-48e9-87ca-8bdf943aa203 None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:56:08.427966 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf50a238-44b5-48e9-87ca-8bdf943aa203 None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:56:08.428240 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf50a238-44b5-48e9-87ca-8bdf943aa203 None admin] PATH_INFO: `/v3/users/9596c91ee75e4abbbe5ac04486da0981` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:56:08.429351 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf50a238-44b5-48e9-87ca-8bdf943aa203 None admin] RBAC: Authorizing `identity:get_user(user_id=9596c91ee75e4abbbe5ac04486da0981)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:56:08.431085 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf50a238-44b5-48e9-87ca-8bdf943aa203 None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:56:08.432650 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 608/1197] 10.222.0.50 () {56 vars in 1177 bytes} [Tue Sep 13 15:56:08 2022] GET /identity/v3/users/9596c91ee75e4abbbe5ac04486da0981 => generated 311 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:56:08.438438 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d55d8e17-4bd4-4d5d-90bf-b8e8caf89ed7 None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:56:08.546269 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d55d8e17-4bd4-4d5d-90bf-b8e8caf89ed7 None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:56:08.578299 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e8267ce-05c3-4dad-b334-5f1fd74bac44 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:56:08.578893 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e8267ce-05c3-4dad-b334-5f1fd74bac44 None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:56:08.579201 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e8267ce-05c3-4dad-b334-5f1fd74bac44 None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:56:08.579472 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e8267ce-05c3-4dad-b334-5f1fd74bac44 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:56:08.632362 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:56:08.644854 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e8267ce-05c3-4dad-b334-5f1fd74bac44 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:56:08.646283 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e8267ce-05c3-4dad-b334-5f1fd74bac44 None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:56:08.740596 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 590/1198] 10.222.0.50 () {58 vars in 1282 bytes} [Tue Sep 13 15:56:08 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 303 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 13 15:56:08.746536 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf50a238-44b5-48e9-87ca-8bdf943aa203 None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:56:08.844458 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf50a238-44b5-48e9-87ca-8bdf943aa203 None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:56:08.878261 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75cfbc75-c0e0-4c8a-ae19-a733efd4137b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:56:08.878882 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75cfbc75-c0e0-4c8a-ae19-a733efd4137b None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:56:08.879197 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75cfbc75-c0e0-4c8a-ae19-a733efd4137b None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:56:08.879471 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75cfbc75-c0e0-4c8a-ae19-a733efd4137b None admin] PATH_INFO: `/v3/projects/642b5b41c00f4463ac985538cd5ef6b9` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:56:08.880581 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75cfbc75-c0e0-4c8a-ae19-a733efd4137b None admin] RBAC: Authorizing `identity:get_project(project_id=642b5b41c00f4463ac985538cd5ef6b9)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:56:08.882019 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75cfbc75-c0e0-4c8a-ae19-a733efd4137b None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:56:08.883687 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 609/1199] 10.222.0.50 () {56 vars in 1186 bytes} [Tue Sep 13 15:56:08 2022] GET /identity/v3/projects/642b5b41c00f4463ac985538cd5ef6b9 => generated 361 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:56:08.889799 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e8267ce-05c3-4dad-b334-5f1fd74bac44 None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:56:08.975831 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:56:08.988494 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e8267ce-05c3-4dad-b334-5f1fd74bac44 None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:56:09.022093 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d992f5a3-6dd2-44c5-a7f3-5060e3e5e9b3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:56:09.022686 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d992f5a3-6dd2-44c5-a7f3-5060e3e5e9b3 None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:56:09.022993 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d992f5a3-6dd2-44c5-a7f3-5060e3e5e9b3 None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:56:09.023268 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d992f5a3-6dd2-44c5-a7f3-5060e3e5e9b3 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:56:09.023976 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d992f5a3-6dd2-44c5-a7f3-5060e3e5e9b3 None admin] RBAC: Adding query filter params (role.id=031d4e1a23e34f71858fc4a6dfbd4ccb, scope.project.id=642b5b41c00f4463ac985538cd5ef6b9, user.id=9596c91ee75e4abbbe5ac04486da0981) {{(pid=81438) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:56:09.024284 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d992f5a3-6dd2-44c5-a7f3-5060e3e5e9b3 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:56:09.025693 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d992f5a3-6dd2-44c5-a7f3-5060e3e5e9b3 None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:56:09.031253 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 591/1200] 10.222.0.50 () {56 vars in 1374 bytes} [Tue Sep 13 15:56:08 2022] GET /identity/v3/role_assignments?user.id=9596c91ee75e4abbbe5ac04486da0981&scope.project.id=642b5b41c00f4463ac985538cd5ef6b9&role.id=031d4e1a23e34f71858fc4a6dfbd4ccb => generated 608 bytes in 142 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:56:18.874018 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=90 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:56:19.028702 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=24 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:57:04.782246 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0254be01-fa42-4002-865c-d5bf3c90d5dc None None] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:57:04.782246 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0254be01-fa42-4002-865c-d5bf3c90d5dc None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:57:04.782246 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0254be01-fa42-4002-865c-d5bf3c90d5dc None None] PATH_INFO: `/` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:57:04.782246 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 610/1201] 10.222.0.50 () {54 vars in 927 bytes} [Tue Sep 13 15:57:04 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:57:04.788625 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d7157c1-301b-496f-81b6-6558ee813ac5 None None] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:57:04.789135 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d7157c1-301b-496f-81b6-6558ee813ac5 None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:57:04.789234 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d7157c1-301b-496f-81b6-6558ee813ac5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:57:04.817367 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.auth.core [None req-5d7157c1-301b-496f-81b6-6558ee813ac5 None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81438) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:57:04.819373 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-5d7157c1-301b-496f-81b6-6558ee813ac5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:57:04.920789 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 592/1202] 10.222.0.50 () {58 vars in 1024 bytes} [Tue Sep 13 15:57:04 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 133 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 13 15:57:05.197196 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0254be01-fa42-4002-865c-d5bf3c90d5dc None None] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:57:05.331440 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0254be01-fa42-4002-865c-d5bf3c90d5dc None None] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:57:05.373711 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1630400-b19d-4de8-bb25-643c51e97c69 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:57:05.373975 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1630400-b19d-4de8-bb25-643c51e97c69 None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:57:05.374585 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1630400-b19d-4de8-bb25-643c51e97c69 None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:57:05.374585 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1630400-b19d-4de8-bb25-643c51e97c69 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:57:05.386393 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1630400-b19d-4de8-bb25-643c51e97c69 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:57:05.387894 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1630400-b19d-4de8-bb25-643c51e97c69 None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:57:05.406410 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application [None req-e1630400-b19d-4de8-bb25-643c51e97c69 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:57:05.406410 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:57:05.406410 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:57:05.406410 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:57:05.406410 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:57:05.406410 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:57:05.406410 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:57:05.406410 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:57:05.406410 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:57:05.406410 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:57:05.406410 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:57:05.406410 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:57:05.406410 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 13 15:57:05.406410 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 13 15:57:05.406410 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 13 15:57:05.406410 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 13 15:57:05.407911 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:57:05.407911 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:57:05.407911 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:57:05.407911 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:57:05.407911 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:57:05.407911 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:57:05.407911 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:57:05.407911 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application with Lock( Sep 13 15:57:05.407911 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:57:05.407911 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:57:05.407911 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:57:05.407911 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:57:05.407911 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:57:05.407911 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:57:05.407911 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:57:05.407911 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:57:05.407911 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 13 15:57:05.409431 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 13 15:57:05.409431 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:57:05.409431 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application Sep 13 15:57:05.413094 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 611/1203] 10.222.0.50 () {56 vars in 1108 bytes} [Tue Sep 13 15:57:05 2022] GET /identity/v3/domains/Default => generated 87 bytes in 211 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:57:05.415074 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d7157c1-301b-496f-81b6-6558ee813ac5 None None] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:57:05.537712 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d7157c1-301b-496f-81b6-6558ee813ac5 None None] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:57:05.571967 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:57:05.578694 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a675410-2a38-4e32-9799-5453657b9a5e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:57:05.579343 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a675410-2a38-4e32-9799-5453657b9a5e None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:57:05.580166 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a675410-2a38-4e32-9799-5453657b9a5e None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:57:05.580445 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a675410-2a38-4e32-9799-5453657b9a5e None admin] PATH_INFO: `/v3/domains` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:57:05.581176 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a675410-2a38-4e32-9799-5453657b9a5e None admin] RBAC: Adding query filter params (name=Default) {{(pid=81438) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:57:05.581614 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a675410-2a38-4e32-9799-5453657b9a5e None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:57:05.583034 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a675410-2a38-4e32-9799-5453657b9a5e None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:57:05.600329 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 593/1204] 10.222.0.50 () {56 vars in 1109 bytes} [Tue Sep 13 15:57:05 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 186 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:57:05.606548 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1630400-b19d-4de8-bb25-643c51e97c69 None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:57:05.672074 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:57:05.730400 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1630400-b19d-4de8-bb25-643c51e97c69 None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:57:05.782083 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2392be57-e17e-4d70-91e4-a933c1262f5e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:57:05.783026 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2392be57-e17e-4d70-91e4-a933c1262f5e None admin] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:57:05.783608 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2392be57-e17e-4d70-91e4-a933c1262f5e None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:57:05.783608 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2392be57-e17e-4d70-91e4-a933c1262f5e None admin] PATH_INFO: `/v3/users` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:57:05.784225 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2392be57-e17e-4d70-91e4-a933c1262f5e None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:57:05.785657 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2392be57-e17e-4d70-91e4-a933c1262f5e None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:57:05.811497 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 612/1205] 10.222.0.50 () {60 vars in 1132 bytes} [Tue Sep 13 15:57:05 2022] POST /identity/v3/users => generated 251 bytes in 205 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 15:57:08.096883 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a7dcb84-3265-4e2b-a677-9031d0b9d757 None None] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:57:08.097425 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a7dcb84-3265-4e2b-a677-9031d0b9d757 None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:57:08.097425 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a7dcb84-3265-4e2b-a677-9031d0b9d757 None None] PATH_INFO: `/` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:57:08.098959 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 594/1206] 10.222.0.50 () {54 vars in 927 bytes} [Tue Sep 13 15:57:08 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:57:08.104132 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c3adc9b-7863-4d93-ab10-d6d8b9c649ed None None] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:57:08.105944 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c3adc9b-7863-4d93-ab10-d6d8b9c649ed None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:57:08.105944 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c3adc9b-7863-4d93-ab10-d6d8b9c649ed None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:57:08.124062 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.auth.core [None req-3c3adc9b-7863-4d93-ab10-d6d8b9c649ed None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81437) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:57:08.125312 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-3c3adc9b-7863-4d93-ab10-d6d8b9c649ed None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:57:08.205043 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 613/1207] 10.222.0.50 () {58 vars in 1024 bytes} [Tue Sep 13 15:57:08 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 13 15:57:08.475412 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a7dcb84-3265-4e2b-a677-9031d0b9d757 None None] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:57:08.598034 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a7dcb84-3265-4e2b-a677-9031d0b9d757 None None] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:57:08.644565 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5079492-a450-44ad-b87f-646ab8219b87 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:57:08.645841 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5079492-a450-44ad-b87f-646ab8219b87 None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:57:08.646130 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5079492-a450-44ad-b87f-646ab8219b87 None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:57:08.646429 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5079492-a450-44ad-b87f-646ab8219b87 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:57:08.660706 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5079492-a450-44ad-b87f-646ab8219b87 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 13 15:57:08.666786 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application [None req-d5079492-a450-44ad-b87f-646ab8219b87 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 13 15:57:08.666786 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:57:08.666786 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:57:08.666786 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:57:08.666786 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:57:08.666786 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:57:08.666786 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:57:08.666786 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:57:08.666786 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:57:08.666786 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:57:08.666786 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:57:08.666786 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:57:08.666786 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 13 15:57:08.666786 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 13 15:57:08.666786 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 13 15:57:08.666786 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 13 15:57:08.669379 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 13 15:57:08.669379 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 13 15:57:08.669379 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 13 15:57:08.669379 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 13 15:57:08.669379 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:57:08.669379 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:57:08.669379 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:57:08.669379 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:57:08.669379 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:57:08.669379 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:57:08.669379 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:57:08.669379 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application with Lock( Sep 13 15:57:08.669379 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:57:08.669379 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:57:08.669379 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:57:08.669379 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:57:08.670262 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:57:08.670262 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:57:08.670262 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:57:08.670262 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:57:08.670262 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 13 15:57:08.670262 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 13 15:57:08.670262 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 13 15:57:08.670262 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 13 15:57:08.670262 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 13 15:57:08.670262 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 13 15:57:08.670262 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 13 15:57:08.670262 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Sep 13 15:57:08.670262 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 595/1208] 10.222.0.50 () {56 vars in 1102 bytes} [Tue Sep 13 15:57:08 2022] GET /identity/v3/roles/service => generated 85 bytes in 193 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 13 15:57:08.674611 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c3adc9b-7863-4d93-ab10-d6d8b9c649ed None None] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:57:08.724998 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:57:08.805555 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c3adc9b-7863-4d93-ab10-d6d8b9c649ed None None] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:57:08.847667 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14b869a1-1290-46d3-a026-434d052ae5d3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:57:08.848345 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14b869a1-1290-46d3-a026-434d052ae5d3 None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:57:08.848676 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14b869a1-1290-46d3-a026-434d052ae5d3 None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:57:08.849197 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14b869a1-1290-46d3-a026-434d052ae5d3 None admin] PATH_INFO: `/v3/roles` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:57:08.849668 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14b869a1-1290-46d3-a026-434d052ae5d3 None admin] RBAC: Adding query filter params (name=service) {{(pid=81437) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:57:08.850000 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14b869a1-1290-46d3-a026-434d052ae5d3 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:57:08.851644 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14b869a1-1290-46d3-a026-434d052ae5d3 None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:57:08.861306 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 614/1209] 10.222.0.50 () {56 vars in 1103 bytes} [Tue Sep 13 15:57:08 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 188 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:57:08.867459 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5079492-a450-44ad-b87f-646ab8219b87 None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:57:08.925858 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:57:08.992881 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5079492-a450-44ad-b87f-646ab8219b87 None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:57:09.032062 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c1fab8d-b6c4-4d1f-8cf8-97a6243c9b29 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:57:09.033594 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c1fab8d-b6c4-4d1f-8cf8-97a6243c9b29 None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:57:09.033918 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c1fab8d-b6c4-4d1f-8cf8-97a6243c9b29 None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:57:09.034230 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c1fab8d-b6c4-4d1f-8cf8-97a6243c9b29 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:57:09.049709 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c1fab8d-b6c4-4d1f-8cf8-97a6243c9b29 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:57:09.051454 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c1fab8d-b6c4-4d1f-8cf8-97a6243c9b29 None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:57:09.071409 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application [None req-3c1fab8d-b6c4-4d1f-8cf8-97a6243c9b29 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:57:09.071409 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:57:09.071409 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:57:09.071409 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:57:09.071409 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:57:09.071409 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:57:09.071409 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:57:09.071409 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:57:09.071409 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:57:09.071409 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:57:09.071409 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:57:09.071409 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:57:09.071409 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 13 15:57:09.071409 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 13 15:57:09.071409 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 13 15:57:09.071409 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 13 15:57:09.072635 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:57:09.072635 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:57:09.072635 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:57:09.072635 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:57:09.072635 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:57:09.072635 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:57:09.072635 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:57:09.072635 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application with Lock( Sep 13 15:57:09.072635 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:57:09.072635 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:57:09.072635 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:57:09.072635 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:57:09.072635 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:57:09.072635 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:57:09.072635 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:57:09.072635 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:57:09.072635 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 13 15:57:09.073826 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 13 15:57:09.073826 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:57:09.073826 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Sep 13 15:57:09.075306 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 596/1210] 10.222.0.50 () {56 vars in 1108 bytes} [Tue Sep 13 15:57:08 2022] GET /identity/v3/domains/Default => generated 87 bytes in 208 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:57:09.078439 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14b869a1-1290-46d3-a026-434d052ae5d3 None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:57:09.199888 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:57:09.203001 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14b869a1-1290-46d3-a026-434d052ae5d3 None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:57:09.244820 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52615f0c-28f8-47a0-a5c7-5a2a7631a355 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:57:09.245921 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52615f0c-28f8-47a0-a5c7-5a2a7631a355 None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:57:09.245921 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52615f0c-28f8-47a0-a5c7-5a2a7631a355 None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:57:09.245921 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52615f0c-28f8-47a0-a5c7-5a2a7631a355 None admin] PATH_INFO: `/v3/domains` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:57:09.246567 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52615f0c-28f8-47a0-a5c7-5a2a7631a355 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81437) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:57:09.246866 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52615f0c-28f8-47a0-a5c7-5a2a7631a355 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:57:09.248264 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52615f0c-28f8-47a0-a5c7-5a2a7631a355 None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:57:09.261124 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 615/1211] 10.222.0.50 () {56 vars in 1109 bytes} [Tue Sep 13 15:57:09 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 183 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:57:09.266730 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c1fab8d-b6c4-4d1f-8cf8-97a6243c9b29 None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:57:09.399261 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c1fab8d-b6c4-4d1f-8cf8-97a6243c9b29 None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:57:09.401452 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:57:09.442591 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3265fcab-0e04-46e2-a2ce-92dfab49319e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:57:09.443293 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3265fcab-0e04-46e2-a2ce-92dfab49319e None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:57:09.443573 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3265fcab-0e04-46e2-a2ce-92dfab49319e None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:57:09.443843 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3265fcab-0e04-46e2-a2ce-92dfab49319e None admin] PATH_INFO: `/v3/users/placement` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:57:09.452210 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3265fcab-0e04-46e2-a2ce-92dfab49319e None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:57:09.454322 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3265fcab-0e04-46e2-a2ce-92dfab49319e None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:57:09.468939 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application [None req-3265fcab-0e04-46e2-a2ce-92dfab49319e None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Sep 13 15:57:09.468939 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:57:09.468939 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:57:09.468939 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:57:09.468939 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:57:09.468939 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:57:09.468939 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:57:09.468939 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:57:09.468939 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:57:09.468939 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:57:09.468939 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:57:09.468939 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:57:09.468939 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 13 15:57:09.468939 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 13 15:57:09.468939 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 13 15:57:09.468939 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 13 15:57:09.470797 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:57:09.470797 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:57:09.470797 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 13 15:57:09.470797 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 13 15:57:09.470797 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 13 15:57:09.470797 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 13 15:57:09.470797 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:57:09.470797 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:57:09.470797 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:57:09.470797 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:57:09.470797 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:57:09.470797 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application with Lock( Sep 13 15:57:09.470797 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:57:09.470797 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:57:09.470797 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:57:09.470797 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:57:09.470797 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:57:09.470797 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:57:09.472103 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:57:09.472103 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:57:09.472103 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 13 15:57:09.472103 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 13 15:57:09.472103 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 13 15:57:09.472103 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 13 15:57:09.472103 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 13 15:57:09.472103 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 13 15:57:09.472103 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Sep 13 15:57:09.472103 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Sep 13 15:57:09.472103 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 597/1212] 10.222.0.50 () {56 vars in 1108 bytes} [Tue Sep 13 15:57:09 2022] GET /identity/v3/users/placement => generated 87 bytes in 205 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:57:09.479543 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52615f0c-28f8-47a0-a5c7-5a2a7631a355 None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:57:09.596956 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52615f0c-28f8-47a0-a5c7-5a2a7631a355 None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:57:09.641650 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6c0d6ca-adb7-42b6-a4b8-046f10036222 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:57:09.643029 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6c0d6ca-adb7-42b6-a4b8-046f10036222 None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:57:09.643404 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6c0d6ca-adb7-42b6-a4b8-046f10036222 None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:57:09.643776 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6c0d6ca-adb7-42b6-a4b8-046f10036222 None admin] PATH_INFO: `/v3/users` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:57:09.644644 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6c0d6ca-adb7-42b6-a4b8-046f10036222 None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=81437) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:57:09.645055 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6c0d6ca-adb7-42b6-a4b8-046f10036222 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:57:09.646627 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6c0d6ca-adb7-42b6-a4b8-046f10036222 None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:57:09.667577 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 616/1213] 10.222.0.50 () {56 vars in 1143 bytes} [Tue Sep 13 15:57:09 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 188 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:57:09.673533 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3265fcab-0e04-46e2-a2ce-92dfab49319e None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:57:09.800081 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3265fcab-0e04-46e2-a2ce-92dfab49319e None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:57:09.842324 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3a0bd91-9462-4e98-9dfc-841b1f40d25b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:57:09.843136 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3a0bd91-9462-4e98-9dfc-841b1f40d25b None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:57:09.843527 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3a0bd91-9462-4e98-9dfc-841b1f40d25b None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:57:09.843932 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3a0bd91-9462-4e98-9dfc-841b1f40d25b None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:57:09.881899 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:57:09.935700 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3a0bd91-9462-4e98-9dfc-841b1f40d25b None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:57:09.937164 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3a0bd91-9462-4e98-9dfc-841b1f40d25b None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:57:10.064409 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 598/1214] 10.222.0.50 () {58 vars in 1282 bytes} [Tue Sep 13 15:57:09 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 384 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 13 15:57:10.064669 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6c0d6ca-adb7-42b6-a4b8-046f10036222 None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:57:10.085633 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:57:10.185324 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6c0d6ca-adb7-42b6-a4b8-046f10036222 None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:57:10.224361 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5997a69-71db-4bb8-9664-c59efdce7269 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:57:10.225010 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5997a69-71db-4bb8-9664-c59efdce7269 None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:57:10.225268 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5997a69-71db-4bb8-9664-c59efdce7269 None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:57:10.225619 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5997a69-71db-4bb8-9664-c59efdce7269 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:57:10.237945 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5997a69-71db-4bb8-9664-c59efdce7269 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:57:10.239427 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5997a69-71db-4bb8-9664-c59efdce7269 None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:57:10.256693 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application [None req-c5997a69-71db-4bb8-9664-c59efdce7269 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:57:10.256693 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:57:10.256693 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:57:10.256693 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:57:10.256693 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:57:10.256693 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:57:10.256693 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:57:10.256693 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:57:10.256693 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:57:10.256693 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:57:10.256693 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:57:10.256693 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:57:10.256693 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 13 15:57:10.256693 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 13 15:57:10.256693 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 13 15:57:10.256693 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 13 15:57:10.257860 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:57:10.257860 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:57:10.257860 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:57:10.257860 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:57:10.257860 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:57:10.257860 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:57:10.257860 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:57:10.257860 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application with Lock( Sep 13 15:57:10.257860 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:57:10.257860 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:57:10.257860 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:57:10.257860 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:57:10.257860 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:57:10.257860 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:57:10.257860 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:57:10.257860 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:57:10.257860 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 13 15:57:10.258792 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 13 15:57:10.258792 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:57:10.258792 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application Sep 13 15:57:10.258792 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 617/1215] 10.222.0.50 () {56 vars in 1108 bytes} [Tue Sep 13 15:57:10 2022] GET /identity/v3/domains/Default => generated 87 bytes in 195 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:57:10.264104 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3a0bd91-9462-4e98-9dfc-841b1f40d25b None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:57:10.338231 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:57:10.383131 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3a0bd91-9462-4e98-9dfc-841b1f40d25b None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:57:10.419323 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd41d631-922d-47b2-b2e3-e3c0375ba015 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:57:10.420097 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd41d631-922d-47b2-b2e3-e3c0375ba015 None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:57:10.420483 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd41d631-922d-47b2-b2e3-e3c0375ba015 None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:57:10.420850 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd41d631-922d-47b2-b2e3-e3c0375ba015 None admin] PATH_INFO: `/v3/domains` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:57:10.421589 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd41d631-922d-47b2-b2e3-e3c0375ba015 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81438) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:57:10.421891 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd41d631-922d-47b2-b2e3-e3c0375ba015 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:57:10.423437 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd41d631-922d-47b2-b2e3-e3c0375ba015 None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:57:10.435810 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 599/1216] 10.222.0.50 () {56 vars in 1109 bytes} [Tue Sep 13 15:57:10 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:57:10.442434 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5997a69-71db-4bb8-9664-c59efdce7269 None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:57:10.520676 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:57:10.557491 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5997a69-71db-4bb8-9664-c59efdce7269 None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:57:10.595016 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-632cffc2-2b90-4a36-8415-923517f79fff None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:57:10.595674 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-632cffc2-2b90-4a36-8415-923517f79fff None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:57:10.595992 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-632cffc2-2b90-4a36-8415-923517f79fff None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:57:10.596186 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-632cffc2-2b90-4a36-8415-923517f79fff None admin] PATH_INFO: `/v3/projects/service` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:57:10.603202 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-632cffc2-2b90-4a36-8415-923517f79fff None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:57:10.604670 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-632cffc2-2b90-4a36-8415-923517f79fff None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:57:10.618098 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application [None req-632cffc2-2b90-4a36-8415-923517f79fff None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 13 15:57:10.618098 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:57:10.618098 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:57:10.618098 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:57:10.618098 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:57:10.618098 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:57:10.618098 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:57:10.618098 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:57:10.618098 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:57:10.618098 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:57:10.618098 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:57:10.618098 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:57:10.618098 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 13 15:57:10.618098 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 13 15:57:10.618098 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 13 15:57:10.619334 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 13 15:57:10.619334 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:57:10.619334 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:57:10.619334 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:57:10.619334 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:57:10.619334 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:57:10.619334 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:57:10.619334 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:57:10.619334 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application with Lock( Sep 13 15:57:10.619334 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:57:10.619334 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:57:10.619334 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:57:10.619334 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:57:10.619334 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:57:10.619334 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:57:10.619334 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:57:10.619334 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:57:10.620321 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 13 15:57:10.620321 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 13 15:57:10.620321 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 13 15:57:10.620321 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 13 15:57:10.620321 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 13 15:57:10.620321 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 13 15:57:10.620321 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 13 15:57:10.620321 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application Sep 13 15:57:10.620321 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 618/1217] 10.222.0.50 () {56 vars in 1111 bytes} [Tue Sep 13 15:57:10 2022] GET /identity/v3/projects/service => generated 88 bytes in 177 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:57:10.625794 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd41d631-922d-47b2-b2e3-e3c0375ba015 None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:57:10.755834 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd41d631-922d-47b2-b2e3-e3c0375ba015 None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:57:10.783415 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:57:10.799664 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9697abba-5662-4b3e-8f46-a0b3ec50d19d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:57:10.800457 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9697abba-5662-4b3e-8f46-a0b3ec50d19d None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:57:10.800806 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9697abba-5662-4b3e-8f46-a0b3ec50d19d None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:57:10.801146 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9697abba-5662-4b3e-8f46-a0b3ec50d19d None admin] PATH_INFO: `/v3/projects` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:57:10.802174 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9697abba-5662-4b3e-8f46-a0b3ec50d19d None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81438) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:57:10.802560 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9697abba-5662-4b3e-8f46-a0b3ec50d19d None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:57:10.804110 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9697abba-5662-4b3e-8f46-a0b3ec50d19d None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:57:10.819362 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 600/1218] 10.222.0.50 () {56 vars in 1148 bytes} [Tue Sep 13 15:57:10 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 194 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:57:10.826244 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-632cffc2-2b90-4a36-8415-923517f79fff None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:57:10.952754 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-632cffc2-2b90-4a36-8415-923517f79fff None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:57:10.987927 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:57:10.995987 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bfb36a3-7f99-4583-b7aa-cc356b977819 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:57:10.996669 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bfb36a3-7f99-4583-b7aa-cc356b977819 None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:57:10.996940 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bfb36a3-7f99-4583-b7aa-cc356b977819 None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:57:10.997217 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bfb36a3-7f99-4583-b7aa-cc356b977819 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:57:10.997939 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bfb36a3-7f99-4583-b7aa-cc356b977819 None admin] RBAC: Adding query filter params (role.id=da847124d9374ac88b27968451c486ab, scope.project.id=5811e3b83a1e462e9320cad79a3ad791, user.id=31480b61418340bd88583f034f67412e) {{(pid=81437) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:57:10.998245 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bfb36a3-7f99-4583-b7aa-cc356b977819 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:57:10.999875 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bfb36a3-7f99-4583-b7aa-cc356b977819 None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:57:11.006182 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 619/1219] 10.222.0.50 () {56 vars in 1374 bytes} [Tue Sep 13 15:57:10 2022] GET /identity/v3/role_assignments?user.id=31480b61418340bd88583f034f67412e&scope.project.id=5811e3b83a1e462e9320cad79a3ad791&role.id=da847124d9374ac88b27968451c486ab => generated 259 bytes in 180 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:57:13.329947 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65ba77b6-ebad-4196-96f8-cad07b6e20d4 None None] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:57:13.330524 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65ba77b6-ebad-4196-96f8-cad07b6e20d4 None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:57:13.330672 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65ba77b6-ebad-4196-96f8-cad07b6e20d4 None None] PATH_INFO: `/` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:57:13.332480 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 601/1220] 10.222.0.50 () {54 vars in 927 bytes} [Tue Sep 13 15:57:13 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:57:13.337476 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b58dd5e7-7209-4b33-b408-fdfd25122a93 None None] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:57:13.337893 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b58dd5e7-7209-4b33-b408-fdfd25122a93 None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:57:13.338035 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b58dd5e7-7209-4b33-b408-fdfd25122a93 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:57:13.357057 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.auth.core [None req-b58dd5e7-7209-4b33-b408-fdfd25122a93 None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81437) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:57:13.360163 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-b58dd5e7-7209-4b33-b408-fdfd25122a93 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:57:13.439695 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 620/1221] 10.222.0.50 () {58 vars in 1024 bytes} [Tue Sep 13 15:57:13 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 13 15:57:13.730360 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65ba77b6-ebad-4196-96f8-cad07b6e20d4 None None] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:57:13.852502 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65ba77b6-ebad-4196-96f8-cad07b6e20d4 None None] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:57:13.888496 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25e85d41-e4b4-4372-9694-d1587945a0bd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:57:13.889143 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25e85d41-e4b4-4372-9694-d1587945a0bd None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:57:13.889426 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25e85d41-e4b4-4372-9694-d1587945a0bd None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:57:13.889707 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25e85d41-e4b4-4372-9694-d1587945a0bd None admin] PATH_INFO: `/v3/roles/service` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:57:13.902278 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25e85d41-e4b4-4372-9694-d1587945a0bd None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 13 15:57:13.907773 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application [None req-25e85d41-e4b4-4372-9694-d1587945a0bd None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 13 15:57:13.907773 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:57:13.907773 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:57:13.907773 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:57:13.907773 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:57:13.907773 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:57:13.907773 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:57:13.907773 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:57:13.907773 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:57:13.907773 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:57:13.907773 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:57:13.907773 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:57:13.907773 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 13 15:57:13.907773 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 13 15:57:13.907773 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 13 15:57:13.907773 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 13 15:57:13.908756 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 13 15:57:13.908756 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 13 15:57:13.908756 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 13 15:57:13.908756 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 13 15:57:13.908756 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:57:13.908756 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:57:13.908756 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:57:13.908756 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:57:13.908756 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:57:13.908756 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:57:13.908756 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:57:13.908756 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application with Lock( Sep 13 15:57:13.908756 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:57:13.908756 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:57:13.908756 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:57:13.908756 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:57:13.909804 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:57:13.909804 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:57:13.909804 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:57:13.909804 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:57:13.909804 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 13 15:57:13.909804 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 13 15:57:13.909804 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 13 15:57:13.909804 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 13 15:57:13.909804 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 13 15:57:13.909804 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 13 15:57:13.909804 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 13 15:57:13.909804 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Sep 13 15:57:13.909804 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 602/1222] 10.222.0.50 () {56 vars in 1102 bytes} [Tue Sep 13 15:57:13 2022] GET /identity/v3/roles/service => generated 85 bytes in 180 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 13 15:57:13.915541 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b58dd5e7-7209-4b33-b408-fdfd25122a93 None None] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:57:14.017006 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b58dd5e7-7209-4b33-b408-fdfd25122a93 None None] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:57:14.040398 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:57:14.053268 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d152ee56-584d-4f58-ba7b-670e2a8da812 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:57:14.053975 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d152ee56-584d-4f58-ba7b-670e2a8da812 None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:57:14.054277 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d152ee56-584d-4f58-ba7b-670e2a8da812 None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:57:14.054546 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d152ee56-584d-4f58-ba7b-670e2a8da812 None admin] PATH_INFO: `/v3/roles` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:57:14.055359 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d152ee56-584d-4f58-ba7b-670e2a8da812 None admin] RBAC: Adding query filter params (name=service) {{(pid=81437) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:57:14.055713 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d152ee56-584d-4f58-ba7b-670e2a8da812 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:57:14.057346 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d152ee56-584d-4f58-ba7b-670e2a8da812 None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:57:14.066927 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 621/1223] 10.222.0.50 () {56 vars in 1103 bytes} [Tue Sep 13 15:57:13 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:57:14.072543 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25e85d41-e4b4-4372-9694-d1587945a0bd None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:57:14.118194 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:57:14.178366 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25e85d41-e4b4-4372-9694-d1587945a0bd None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:57:14.213710 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdbe5b20-a355-4361-9a5e-1d5422ad1e3b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:57:14.214337 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdbe5b20-a355-4361-9a5e-1d5422ad1e3b None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:57:14.214609 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdbe5b20-a355-4361-9a5e-1d5422ad1e3b None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:57:14.214877 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdbe5b20-a355-4361-9a5e-1d5422ad1e3b None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:57:14.226616 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdbe5b20-a355-4361-9a5e-1d5422ad1e3b None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:57:14.228299 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdbe5b20-a355-4361-9a5e-1d5422ad1e3b None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:57:14.244234 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application [None req-bdbe5b20-a355-4361-9a5e-1d5422ad1e3b None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:57:14.244234 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:57:14.244234 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:57:14.244234 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:57:14.244234 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:57:14.244234 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:57:14.244234 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:57:14.244234 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:57:14.244234 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:57:14.244234 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:57:14.244234 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:57:14.244234 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:57:14.244234 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 13 15:57:14.244234 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 13 15:57:14.244234 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 13 15:57:14.244234 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 13 15:57:14.245435 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:57:14.245435 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:57:14.245435 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:57:14.245435 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:57:14.245435 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:57:14.245435 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:57:14.245435 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:57:14.245435 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application with Lock( Sep 13 15:57:14.245435 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:57:14.245435 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:57:14.245435 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:57:14.245435 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:57:14.245435 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:57:14.245435 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:57:14.245435 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:57:14.245435 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:57:14.245435 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 13 15:57:14.246461 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 13 15:57:14.246461 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:57:14.246461 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Sep 13 15:57:14.246461 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 603/1224] 10.222.0.50 () {56 vars in 1108 bytes} [Tue Sep 13 15:57:14 2022] GET /identity/v3/domains/Default => generated 87 bytes in 174 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:57:14.252908 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d152ee56-584d-4f58-ba7b-670e2a8da812 None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:57:14.358041 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d152ee56-584d-4f58-ba7b-670e2a8da812 None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:57:14.395101 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cf7c106-9b5a-4fe5-95f1-78df44af6954 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:57:14.395747 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cf7c106-9b5a-4fe5-95f1-78df44af6954 None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:57:14.396071 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cf7c106-9b5a-4fe5-95f1-78df44af6954 None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:57:14.396350 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cf7c106-9b5a-4fe5-95f1-78df44af6954 None admin] PATH_INFO: `/v3/domains` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:57:14.397074 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cf7c106-9b5a-4fe5-95f1-78df44af6954 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81437) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:57:14.397418 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cf7c106-9b5a-4fe5-95f1-78df44af6954 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:57:14.398853 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cf7c106-9b5a-4fe5-95f1-78df44af6954 None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:57:14.412288 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 622/1225] 10.222.0.50 () {56 vars in 1109 bytes} [Tue Sep 13 15:57:14 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:57:14.418069 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdbe5b20-a355-4361-9a5e-1d5422ad1e3b None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:57:14.523063 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:57:14.530956 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdbe5b20-a355-4361-9a5e-1d5422ad1e3b None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:57:14.569013 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3b5aaee-70c1-42a8-83ec-c62c70c9f3b6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:57:14.569652 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3b5aaee-70c1-42a8-83ec-c62c70c9f3b6 None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:57:14.569934 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3b5aaee-70c1-42a8-83ec-c62c70c9f3b6 None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:57:14.570300 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3b5aaee-70c1-42a8-83ec-c62c70c9f3b6 None admin] PATH_INFO: `/v3/users/placement` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:57:14.578432 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3b5aaee-70c1-42a8-83ec-c62c70c9f3b6 None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:57:14.580621 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3b5aaee-70c1-42a8-83ec-c62c70c9f3b6 None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:57:14.592586 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application [None req-d3b5aaee-70c1-42a8-83ec-c62c70c9f3b6 None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Sep 13 15:57:14.592586 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:57:14.592586 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:57:14.592586 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:57:14.592586 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:57:14.592586 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:57:14.592586 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:57:14.592586 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:57:14.592586 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:57:14.592586 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:57:14.592586 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:57:14.592586 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:57:14.592586 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 13 15:57:14.592586 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 13 15:57:14.592586 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 13 15:57:14.592586 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 13 15:57:14.593667 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:57:14.593667 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:57:14.593667 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 13 15:57:14.593667 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 13 15:57:14.593667 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 13 15:57:14.593667 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 13 15:57:14.593667 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:57:14.593667 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:57:14.593667 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:57:14.593667 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:57:14.593667 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:57:14.593667 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application with Lock( Sep 13 15:57:14.593667 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:57:14.593667 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:57:14.593667 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:57:14.593667 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:57:14.593667 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:57:14.593667 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:57:14.595353 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:57:14.595353 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:57:14.595353 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 13 15:57:14.595353 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 13 15:57:14.595353 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 13 15:57:14.595353 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 13 15:57:14.595353 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 13 15:57:14.595353 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 13 15:57:14.595353 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Sep 13 15:57:14.595353 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Sep 13 15:57:14.595353 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 604/1226] 10.222.0.50 () {56 vars in 1108 bytes} [Tue Sep 13 15:57:14 2022] GET /identity/v3/users/placement => generated 87 bytes in 176 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:57:14.600676 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cf7c106-9b5a-4fe5-95f1-78df44af6954 None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:57:14.653360 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:57:14.711688 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cf7c106-9b5a-4fe5-95f1-78df44af6954 None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:57:14.751653 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-167d7508-3c90-4ba5-9ff9-e5a058254548 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:57:14.752487 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-167d7508-3c90-4ba5-9ff9-e5a058254548 None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:57:14.752767 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-167d7508-3c90-4ba5-9ff9-e5a058254548 None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:57:14.753039 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-167d7508-3c90-4ba5-9ff9-e5a058254548 None admin] PATH_INFO: `/v3/users` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:57:14.753906 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-167d7508-3c90-4ba5-9ff9-e5a058254548 None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=81437) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:57:14.754294 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-167d7508-3c90-4ba5-9ff9-e5a058254548 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:57:14.755911 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-167d7508-3c90-4ba5-9ff9-e5a058254548 None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:57:14.778233 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 623/1227] 10.222.0.50 () {56 vars in 1143 bytes} [Tue Sep 13 15:57:14 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:57:14.784726 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3b5aaee-70c1-42a8-83ec-c62c70c9f3b6 None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:57:14.896256 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3b5aaee-70c1-42a8-83ec-c62c70c9f3b6 None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:57:14.932308 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9acc603a-d3e6-4953-a343-9ea683dec446 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:57:14.932861 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9acc603a-d3e6-4953-a343-9ea683dec446 None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:57:14.933133 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9acc603a-d3e6-4953-a343-9ea683dec446 None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:57:14.933397 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9acc603a-d3e6-4953-a343-9ea683dec446 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:57:14.945912 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9acc603a-d3e6-4953-a343-9ea683dec446 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:57:14.947547 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9acc603a-d3e6-4953-a343-9ea683dec446 None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:57:14.963297 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application [None req-9acc603a-d3e6-4953-a343-9ea683dec446 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:57:14.963297 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:57:14.963297 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:57:14.963297 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:57:14.963297 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:57:14.963297 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:57:14.963297 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:57:14.963297 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:57:14.963297 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:57:14.963297 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:57:14.963297 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:57:14.963297 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:57:14.963297 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 13 15:57:14.963297 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 13 15:57:14.963297 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 13 15:57:14.963297 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 13 15:57:14.964353 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:57:14.964353 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:57:14.964353 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:57:14.964353 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:57:14.964353 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:57:14.964353 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:57:14.964353 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:57:14.964353 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application with Lock( Sep 13 15:57:14.964353 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:57:14.964353 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:57:14.964353 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:57:14.964353 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:57:14.964353 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:57:14.964353 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:57:14.964353 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:57:14.964353 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:57:14.964353 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 13 15:57:14.965665 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 13 15:57:14.965665 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:57:14.965665 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Sep 13 15:57:14.965665 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 605/1228] 10.222.0.50 () {56 vars in 1108 bytes} [Tue Sep 13 15:57:14 2022] GET /identity/v3/domains/Default => generated 87 bytes in 180 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:57:14.971198 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-167d7508-3c90-4ba5-9ff9-e5a058254548 None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:57:15.089303 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-167d7508-3c90-4ba5-9ff9-e5a058254548 None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:57:15.091763 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:57:15.126201 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7028b3de-53a0-4951-b6b8-b7ab701c292e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:57:15.126814 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7028b3de-53a0-4951-b6b8-b7ab701c292e None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:57:15.127116 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7028b3de-53a0-4951-b6b8-b7ab701c292e None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:57:15.127381 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7028b3de-53a0-4951-b6b8-b7ab701c292e None admin] PATH_INFO: `/v3/domains` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:57:15.128014 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7028b3de-53a0-4951-b6b8-b7ab701c292e None admin] RBAC: Adding query filter params (name=Default) {{(pid=81437) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:57:15.128312 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7028b3de-53a0-4951-b6b8-b7ab701c292e None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:57:15.129667 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7028b3de-53a0-4951-b6b8-b7ab701c292e None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:57:15.140679 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 624/1229] 10.222.0.50 () {56 vars in 1109 bytes} [Tue Sep 13 15:57:14 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 13 15:57:15.146438 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9acc603a-d3e6-4953-a343-9ea683dec446 None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:57:15.155626 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:57:15.252103 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9acc603a-d3e6-4953-a343-9ea683dec446 None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:57:15.288028 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28f11160-d96c-4e17-90c3-940ddf9c16c7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:57:15.288720 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28f11160-d96c-4e17-90c3-940ddf9c16c7 None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:57:15.289039 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28f11160-d96c-4e17-90c3-940ddf9c16c7 None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:57:15.289335 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28f11160-d96c-4e17-90c3-940ddf9c16c7 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:57:15.296477 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28f11160-d96c-4e17-90c3-940ddf9c16c7 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:57:15.298441 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28f11160-d96c-4e17-90c3-940ddf9c16c7 None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:57:15.310188 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application [None req-28f11160-d96c-4e17-90c3-940ddf9c16c7 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 13 15:57:15.310188 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:57:15.310188 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:57:15.310188 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:57:15.310188 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:57:15.310188 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:57:15.310188 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:57:15.310188 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:57:15.310188 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:57:15.310188 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:57:15.310188 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:57:15.310188 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:57:15.310188 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 13 15:57:15.310188 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 13 15:57:15.310188 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 13 15:57:15.311598 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 13 15:57:15.311598 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:57:15.311598 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:57:15.311598 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:57:15.311598 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:57:15.311598 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:57:15.311598 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:57:15.311598 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:57:15.311598 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application with Lock( Sep 13 15:57:15.311598 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:57:15.311598 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:57:15.311598 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:57:15.311598 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:57:15.311598 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:57:15.311598 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:57:15.311598 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:57:15.311598 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:57:15.312871 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 13 15:57:15.312871 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 13 15:57:15.312871 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 13 15:57:15.312871 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 13 15:57:15.312871 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 13 15:57:15.312871 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 13 15:57:15.312871 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 13 15:57:15.312871 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Sep 13 15:57:15.312871 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 606/1230] 10.222.0.50 () {56 vars in 1111 bytes} [Tue Sep 13 15:57:15 2022] GET /identity/v3/projects/service => generated 88 bytes in 165 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:57:15.318512 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7028b3de-53a0-4951-b6b8-b7ab701c292e None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:57:15.435125 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7028b3de-53a0-4951-b6b8-b7ab701c292e None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:57:15.476172 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d01846b7-1c91-49ba-a0a2-3a8f285f8344 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:57:15.476726 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d01846b7-1c91-49ba-a0a2-3a8f285f8344 None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:57:15.477009 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d01846b7-1c91-49ba-a0a2-3a8f285f8344 None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:57:15.477287 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d01846b7-1c91-49ba-a0a2-3a8f285f8344 None admin] PATH_INFO: `/v3/projects` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:57:15.477977 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d01846b7-1c91-49ba-a0a2-3a8f285f8344 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81437) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:57:15.478357 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d01846b7-1c91-49ba-a0a2-3a8f285f8344 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:57:15.480136 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d01846b7-1c91-49ba-a0a2-3a8f285f8344 None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:57:15.494801 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 625/1231] 10.222.0.50 () {56 vars in 1148 bytes} [Tue Sep 13 15:57:15 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:57:15.501297 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28f11160-d96c-4e17-90c3-940ddf9c16c7 None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:57:15.576145 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:57:15.614752 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28f11160-d96c-4e17-90c3-940ddf9c16c7 None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:57:15.650598 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d61b993-3330-4b5c-b102-80e50e69a7c2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:57:15.651285 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d61b993-3330-4b5c-b102-80e50e69a7c2 None admin] REQUEST_METHOD: `PUT` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:57:15.651563 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d61b993-3330-4b5c-b102-80e50e69a7c2 None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:57:15.651835 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d61b993-3330-4b5c-b102-80e50e69a7c2 None admin] PATH_INFO: `/v3/projects/5811e3b83a1e462e9320cad79a3ad791/users/31480b61418340bd88583f034f67412e/roles/da847124d9374ac88b27968451c486ab` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:57:15.667045 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d61b993-3330-4b5c-b102-80e50e69a7c2 None admin] RBAC: Authorizing `identity:create_grant(project_id=5811e3b83a1e462e9320cad79a3ad791, user_id=31480b61418340bd88583f034f67412e, role_id=da847124d9374ac88b27968451c486ab)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:57:15.668509 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d61b993-3330-4b5c-b102-80e50e69a7c2 None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:57:15.681550 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 607/1232] 10.222.0.50 () {58 vars in 1439 bytes} [Tue Sep 13 15:57:15 2022] PUT /identity/v3/projects/5811e3b83a1e462e9320cad79a3ad791/users/31480b61418340bd88583f034f67412e/roles/da847124d9374ac88b27968451c486ab => generated 0 bytes in 181 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 13 15:57:17.845315 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56c6276e-1a54-492f-b024-767809418629 None None] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:57:17.846337 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56c6276e-1a54-492f-b024-767809418629 None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:57:17.846337 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56c6276e-1a54-492f-b024-767809418629 None None] PATH_INFO: `/` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:57:17.847231 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 626/1233] 10.222.0.50 () {54 vars in 927 bytes} [Tue Sep 13 15:57:17 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:57:17.853366 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff3d1320-6c4c-4566-bdec-acc06598c8ea None None] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:57:17.853842 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff3d1320-6c4c-4566-bdec-acc06598c8ea None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:57:17.854311 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff3d1320-6c4c-4566-bdec-acc06598c8ea None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:57:17.880649 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.auth.core [None req-ff3d1320-6c4c-4566-bdec-acc06598c8ea None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81438) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:57:17.883043 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-ff3d1320-6c4c-4566-bdec-acc06598c8ea None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:57:17.968293 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 608/1234] 10.222.0.50 () {58 vars in 1024 bytes} [Tue Sep 13 15:57:17 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 117 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 13 15:57:18.293562 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56c6276e-1a54-492f-b024-767809418629 None None] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:57:18.330481 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:57:18.434078 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56c6276e-1a54-492f-b024-767809418629 None None] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:57:18.477953 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68a0c038-200a-4f62-ba8e-cf3f3ec3778d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:57:18.478611 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68a0c038-200a-4f62-ba8e-cf3f3ec3778d None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:57:18.479043 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68a0c038-200a-4f62-ba8e-cf3f3ec3778d None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:57:18.479318 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68a0c038-200a-4f62-ba8e-cf3f3ec3778d None admin] PATH_INFO: `/v3/roles/service` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:57:18.493865 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68a0c038-200a-4f62-ba8e-cf3f3ec3778d None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 13 15:57:18.501292 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application [None req-68a0c038-200a-4f62-ba8e-cf3f3ec3778d None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 13 15:57:18.501292 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:57:18.501292 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:57:18.501292 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:57:18.501292 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:57:18.501292 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:57:18.501292 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:57:18.501292 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:57:18.501292 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:57:18.501292 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:57:18.501292 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:57:18.501292 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:57:18.501292 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 13 15:57:18.501292 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 13 15:57:18.501292 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 13 15:57:18.501292 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 13 15:57:18.502577 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 13 15:57:18.502577 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 13 15:57:18.502577 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 13 15:57:18.502577 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 13 15:57:18.502577 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:57:18.502577 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:57:18.502577 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:57:18.502577 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:57:18.502577 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:57:18.502577 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:57:18.502577 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:57:18.502577 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application with Lock( Sep 13 15:57:18.502577 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:57:18.502577 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:57:18.502577 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:57:18.502577 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:57:18.503678 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:57:18.503678 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:57:18.503678 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:57:18.503678 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:57:18.503678 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 13 15:57:18.503678 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 13 15:57:18.503678 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 13 15:57:18.503678 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 13 15:57:18.503678 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 13 15:57:18.503678 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 13 15:57:18.503678 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 13 15:57:18.503678 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application Sep 13 15:57:18.507461 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 627/1235] 10.222.0.50 () {56 vars in 1102 bytes} [Tue Sep 13 15:57:18 2022] GET /identity/v3/roles/service => generated 85 bytes in 215 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:57:18.508875 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff3d1320-6c4c-4566-bdec-acc06598c8ea None None] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:57:18.568990 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:57:18.634495 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff3d1320-6c4c-4566-bdec-acc06598c8ea None None] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:57:18.673233 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68dfe6ec-230d-4197-8ba3-5d2cdb7f9e2e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:57:18.673845 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68dfe6ec-230d-4197-8ba3-5d2cdb7f9e2e None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:57:18.674115 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68dfe6ec-230d-4197-8ba3-5d2cdb7f9e2e None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:57:18.674381 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68dfe6ec-230d-4197-8ba3-5d2cdb7f9e2e None admin] PATH_INFO: `/v3/roles` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:57:18.675075 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68dfe6ec-230d-4197-8ba3-5d2cdb7f9e2e None admin] RBAC: Adding query filter params (name=service) {{(pid=81438) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:57:18.675413 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68dfe6ec-230d-4197-8ba3-5d2cdb7f9e2e None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:57:18.677247 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68dfe6ec-230d-4197-8ba3-5d2cdb7f9e2e None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:57:18.689233 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 609/1236] 10.222.0.50 () {56 vars in 1103 bytes} [Tue Sep 13 15:57:18 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:57:18.695976 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68a0c038-200a-4f62-ba8e-cf3f3ec3778d None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:57:18.806850 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:57:18.827205 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68a0c038-200a-4f62-ba8e-cf3f3ec3778d None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:57:18.865867 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a192886-a42a-4247-a3e4-96be7fbe3a1a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:57:18.866550 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a192886-a42a-4247-a3e4-96be7fbe3a1a None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:57:18.866772 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a192886-a42a-4247-a3e4-96be7fbe3a1a None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:57:18.867073 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a192886-a42a-4247-a3e4-96be7fbe3a1a None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:57:18.881190 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a192886-a42a-4247-a3e4-96be7fbe3a1a None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:57:18.882681 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a192886-a42a-4247-a3e4-96be7fbe3a1a None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:57:18.900933 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application [None req-9a192886-a42a-4247-a3e4-96be7fbe3a1a None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:57:18.900933 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:57:18.900933 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:57:18.900933 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:57:18.900933 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:57:18.900933 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:57:18.900933 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:57:18.900933 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:57:18.900933 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:57:18.900933 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:57:18.900933 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:57:18.900933 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:57:18.900933 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 13 15:57:18.900933 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 13 15:57:18.900933 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 13 15:57:18.900933 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 13 15:57:18.902280 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:57:18.902280 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:57:18.902280 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:57:18.902280 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:57:18.902280 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:57:18.902280 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:57:18.902280 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:57:18.902280 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application with Lock( Sep 13 15:57:18.902280 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:57:18.902280 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:57:18.902280 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:57:18.902280 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:57:18.902280 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:57:18.902280 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:57:18.902280 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:57:18.902280 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:57:18.902280 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 13 15:57:18.903285 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 13 15:57:18.903285 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:57:18.903285 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application Sep 13 15:57:18.903285 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 628/1237] 10.222.0.50 () {56 vars in 1108 bytes} [Tue Sep 13 15:57:18 2022] GET /identity/v3/domains/Default => generated 87 bytes in 207 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:57:18.910742 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68dfe6ec-230d-4197-8ba3-5d2cdb7f9e2e None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:57:19.042319 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68dfe6ec-230d-4197-8ba3-5d2cdb7f9e2e None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:57:19.052123 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:57:19.082885 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb23b905-2d3d-4731-a2cf-cee3c38fbb90 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:57:19.083483 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb23b905-2d3d-4731-a2cf-cee3c38fbb90 None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:57:19.083788 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb23b905-2d3d-4731-a2cf-cee3c38fbb90 None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:57:19.084076 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb23b905-2d3d-4731-a2cf-cee3c38fbb90 None admin] PATH_INFO: `/v3/domains` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:57:19.084712 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb23b905-2d3d-4731-a2cf-cee3c38fbb90 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81438) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:57:19.085000 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb23b905-2d3d-4731-a2cf-cee3c38fbb90 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:57:19.086809 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb23b905-2d3d-4731-a2cf-cee3c38fbb90 None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:57:19.111107 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 610/1238] 10.222.0.50 () {56 vars in 1109 bytes} [Tue Sep 13 15:57:18 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 200 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:57:19.111381 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a192886-a42a-4247-a3e4-96be7fbe3a1a None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:57:19.225144 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a192886-a42a-4247-a3e4-96be7fbe3a1a None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:57:19.260970 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04dd72a2-890b-4746-8638-4202fdad0a6b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:57:19.261589 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04dd72a2-890b-4746-8638-4202fdad0a6b None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:57:19.261865 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04dd72a2-890b-4746-8638-4202fdad0a6b None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:57:19.262134 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04dd72a2-890b-4746-8638-4202fdad0a6b None admin] PATH_INFO: `/v3/users/placement` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:57:19.265177 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:57:19.272309 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04dd72a2-890b-4746-8638-4202fdad0a6b None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:57:19.274106 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04dd72a2-890b-4746-8638-4202fdad0a6b None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:57:19.286060 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application [None req-04dd72a2-890b-4746-8638-4202fdad0a6b None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Sep 13 15:57:19.286060 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:57:19.286060 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:57:19.286060 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:57:19.286060 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:57:19.286060 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:57:19.286060 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:57:19.286060 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:57:19.286060 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:57:19.286060 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:57:19.286060 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:57:19.286060 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:57:19.286060 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 13 15:57:19.286060 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 13 15:57:19.286060 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 13 15:57:19.286060 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 13 15:57:19.287429 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:57:19.287429 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:57:19.287429 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 13 15:57:19.287429 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 13 15:57:19.287429 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 13 15:57:19.287429 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 13 15:57:19.287429 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:57:19.287429 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:57:19.287429 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:57:19.287429 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:57:19.287429 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:57:19.287429 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application with Lock( Sep 13 15:57:19.287429 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:57:19.287429 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:57:19.287429 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:57:19.287429 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:57:19.287429 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:57:19.287429 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:57:19.288604 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:57:19.288604 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:57:19.288604 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 13 15:57:19.288604 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 13 15:57:19.288604 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 13 15:57:19.288604 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 13 15:57:19.288604 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 13 15:57:19.288604 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 13 15:57:19.288604 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Sep 13 15:57:19.288604 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application Sep 13 15:57:19.288604 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 629/1239] 10.222.0.50 () {56 vars in 1108 bytes} [Tue Sep 13 15:57:19 2022] GET /identity/v3/users/placement => generated 87 bytes in 177 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:57:19.293887 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb23b905-2d3d-4731-a2cf-cee3c38fbb90 None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:57:19.401861 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb23b905-2d3d-4731-a2cf-cee3c38fbb90 None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:57:19.439494 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8055cb5-cd4c-4f78-920d-187ccddfe2ed None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:57:19.440056 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8055cb5-cd4c-4f78-920d-187ccddfe2ed None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:57:19.440339 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8055cb5-cd4c-4f78-920d-187ccddfe2ed None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:57:19.440611 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8055cb5-cd4c-4f78-920d-187ccddfe2ed None admin] PATH_INFO: `/v3/users` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:57:19.441339 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8055cb5-cd4c-4f78-920d-187ccddfe2ed None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=81438) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:57:19.441643 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8055cb5-cd4c-4f78-920d-187ccddfe2ed None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:57:19.443092 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8055cb5-cd4c-4f78-920d-187ccddfe2ed None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:57:19.461717 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 611/1240] 10.222.0.50 () {56 vars in 1143 bytes} [Tue Sep 13 15:57:19 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 13 15:57:19.467903 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04dd72a2-890b-4746-8638-4202fdad0a6b None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:57:19.576590 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04dd72a2-890b-4746-8638-4202fdad0a6b None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:57:19.615556 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94020468-5189-490e-b7b6-ad63a3daf7fc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:57:19.616165 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94020468-5189-490e-b7b6-ad63a3daf7fc None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:57:19.616462 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94020468-5189-490e-b7b6-ad63a3daf7fc None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:57:19.616785 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94020468-5189-490e-b7b6-ad63a3daf7fc None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:57:19.682833 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:57:19.690624 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94020468-5189-490e-b7b6-ad63a3daf7fc None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:57:19.692200 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94020468-5189-490e-b7b6-ad63a3daf7fc None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:57:19.805751 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 630/1241] 10.222.0.50 () {58 vars in 1282 bytes} [Tue Sep 13 15:57:19 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 338 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 13 15:57:19.812133 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8055cb5-cd4c-4f78-920d-187ccddfe2ed None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:57:19.841757 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:57:19.926920 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8055cb5-cd4c-4f78-920d-187ccddfe2ed None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:57:19.963912 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37924cc0-cc38-44a8-be03-afad0ac79c04 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:57:19.964504 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37924cc0-cc38-44a8-be03-afad0ac79c04 None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:57:19.964784 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37924cc0-cc38-44a8-be03-afad0ac79c04 None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:57:19.965055 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37924cc0-cc38-44a8-be03-afad0ac79c04 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:57:19.976234 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37924cc0-cc38-44a8-be03-afad0ac79c04 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:57:19.977667 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37924cc0-cc38-44a8-be03-afad0ac79c04 None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:57:19.992740 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application [None req-37924cc0-cc38-44a8-be03-afad0ac79c04 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:57:19.992740 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:57:19.992740 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:57:19.992740 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:57:19.992740 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:57:19.992740 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:57:19.992740 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:57:19.992740 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:57:19.992740 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:57:19.992740 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:57:19.992740 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:57:19.992740 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:57:19.992740 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 13 15:57:19.992740 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 13 15:57:19.992740 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 13 15:57:19.992740 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 13 15:57:19.993965 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:57:19.993965 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:57:19.993965 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:57:19.993965 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:57:19.993965 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:57:19.993965 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:57:19.993965 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:57:19.993965 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application with Lock( Sep 13 15:57:19.993965 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:57:19.993965 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:57:19.993965 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:57:19.993965 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:57:19.993965 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:57:19.993965 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:57:19.993965 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:57:19.993965 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:57:19.993965 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 13 15:57:19.994884 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 13 15:57:19.994884 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:57:19.994884 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Sep 13 15:57:19.994884 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 612/1242] 10.222.0.50 () {56 vars in 1108 bytes} [Tue Sep 13 15:57:19 2022] GET /identity/v3/domains/Default => generated 87 bytes in 183 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:57:20.000084 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94020468-5189-490e-b7b6-ad63a3daf7fc None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:57:20.104186 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:57:20.111606 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94020468-5189-490e-b7b6-ad63a3daf7fc None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:57:20.145605 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e20f15a-00f0-4c94-9b5a-bdc5f08ad8fa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:57:20.146313 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e20f15a-00f0-4c94-9b5a-bdc5f08ad8fa None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:57:20.146601 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e20f15a-00f0-4c94-9b5a-bdc5f08ad8fa None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:57:20.146880 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e20f15a-00f0-4c94-9b5a-bdc5f08ad8fa None admin] PATH_INFO: `/v3/domains` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:57:20.147688 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e20f15a-00f0-4c94-9b5a-bdc5f08ad8fa None admin] RBAC: Adding query filter params (name=Default) {{(pid=81437) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:57:20.147982 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e20f15a-00f0-4c94-9b5a-bdc5f08ad8fa None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:57:20.149525 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e20f15a-00f0-4c94-9b5a-bdc5f08ad8fa None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:57:20.162120 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 631/1243] 10.222.0.50 () {56 vars in 1109 bytes} [Tue Sep 13 15:57:19 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:57:20.168101 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37924cc0-cc38-44a8-be03-afad0ac79c04 None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:57:20.259567 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:57:20.286786 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37924cc0-cc38-44a8-be03-afad0ac79c04 None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:57:20.323733 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09030354-b517-41c3-8978-ba6a9d0426e5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:57:20.324355 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09030354-b517-41c3-8978-ba6a9d0426e5 None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:57:20.324632 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09030354-b517-41c3-8978-ba6a9d0426e5 None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:57:20.324904 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09030354-b517-41c3-8978-ba6a9d0426e5 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:57:20.333038 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09030354-b517-41c3-8978-ba6a9d0426e5 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:57:20.334493 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09030354-b517-41c3-8978-ba6a9d0426e5 None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:57:20.346091 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application [None req-09030354-b517-41c3-8978-ba6a9d0426e5 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 13 15:57:20.346091 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:57:20.346091 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:57:20.346091 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:57:20.346091 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:57:20.346091 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:57:20.346091 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:57:20.346091 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:57:20.346091 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:57:20.346091 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:57:20.346091 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:57:20.346091 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:57:20.346091 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 13 15:57:20.346091 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 13 15:57:20.346091 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 13 15:57:20.347330 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 13 15:57:20.347330 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:57:20.347330 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:57:20.347330 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:57:20.347330 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:57:20.347330 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:57:20.347330 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:57:20.347330 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:57:20.347330 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application with Lock( Sep 13 15:57:20.347330 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:57:20.347330 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:57:20.347330 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:57:20.347330 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:57:20.347330 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:57:20.347330 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:57:20.347330 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:57:20.347330 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:57:20.348235 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 13 15:57:20.348235 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 13 15:57:20.348235 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 13 15:57:20.348235 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 13 15:57:20.348235 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 13 15:57:20.348235 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 13 15:57:20.348235 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 13 15:57:20.348235 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Sep 13 15:57:20.348235 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 613/1244] 10.222.0.50 () {56 vars in 1111 bytes} [Tue Sep 13 15:57:20 2022] GET /identity/v3/projects/service => generated 88 bytes in 180 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:57:20.353139 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e20f15a-00f0-4c94-9b5a-bdc5f08ad8fa None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:57:20.471440 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e20f15a-00f0-4c94-9b5a-bdc5f08ad8fa None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:57:20.512482 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13342628-c488-4133-98af-9f12056a2eb2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:57:20.512482 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13342628-c488-4133-98af-9f12056a2eb2 None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:57:20.512813 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13342628-c488-4133-98af-9f12056a2eb2 None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:57:20.513175 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13342628-c488-4133-98af-9f12056a2eb2 None admin] PATH_INFO: `/v3/projects` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:57:20.513899 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13342628-c488-4133-98af-9f12056a2eb2 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81437) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:57:20.514212 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13342628-c488-4133-98af-9f12056a2eb2 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:57:20.515690 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13342628-c488-4133-98af-9f12056a2eb2 None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:57:20.529164 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 632/1245] 10.222.0.50 () {56 vars in 1148 bytes} [Tue Sep 13 15:57:20 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:57:20.535486 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09030354-b517-41c3-8978-ba6a9d0426e5 None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:57:20.660515 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09030354-b517-41c3-8978-ba6a9d0426e5 None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:57:20.706626 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df63c7e3-4716-4606-8758-8d2b8503599a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:57:20.707330 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df63c7e3-4716-4606-8758-8d2b8503599a None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:57:20.707587 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df63c7e3-4716-4606-8758-8d2b8503599a None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:57:20.707938 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df63c7e3-4716-4606-8758-8d2b8503599a None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:57:20.708843 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df63c7e3-4716-4606-8758-8d2b8503599a None admin] RBAC: Adding query filter params (role.id=da847124d9374ac88b27968451c486ab, scope.project.id=5811e3b83a1e462e9320cad79a3ad791, user.id=31480b61418340bd88583f034f67412e) {{(pid=81438) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:57:20.709243 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df63c7e3-4716-4606-8758-8d2b8503599a None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:57:20.710999 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df63c7e3-4716-4606-8758-8d2b8503599a None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:57:20.718588 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 614/1246] 10.222.0.50 () {56 vars in 1374 bytes} [Tue Sep 13 15:57:20 2022] GET /identity/v3/role_assignments?user.id=31480b61418340bd88583f034f67412e&scope.project.id=5811e3b83a1e462e9320cad79a3ad791&role.id=da847124d9374ac88b27968451c486ab => generated 608 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:57:23.178824 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1eba42c8-1f0c-4d96-8a96-c75cf036aff9 None None] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:57:23.179404 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1eba42c8-1f0c-4d96-8a96-c75cf036aff9 None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:57:23.179404 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1eba42c8-1f0c-4d96-8a96-c75cf036aff9 None None] PATH_INFO: `/` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:57:23.180747 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 633/1247] 10.222.0.50 () {54 vars in 927 bytes} [Tue Sep 13 15:57:23 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:57:23.186761 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bfa8a93-e964-4c46-9899-5edfb6c0f209 None None] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:57:23.187236 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bfa8a93-e964-4c46-9899-5edfb6c0f209 None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:57:23.187535 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bfa8a93-e964-4c46-9899-5edfb6c0f209 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:57:23.195462 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:57:23.208631 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.auth.core [None req-5bfa8a93-e964-4c46-9899-5edfb6c0f209 None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81438) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:57:23.209820 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-5bfa8a93-e964-4c46-9899-5edfb6c0f209 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:57:23.281838 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 615/1248] 10.222.0.50 () {58 vars in 1024 bytes} [Tue Sep 13 15:57:23 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 13 15:57:23.569660 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1eba42c8-1f0c-4d96-8a96-c75cf036aff9 None None] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:57:23.594337 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:57:23.684242 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1eba42c8-1f0c-4d96-8a96-c75cf036aff9 None None] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:57:23.723397 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e75fa29-31a8-4d98-b568-9f9a7a48a844 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:57:23.724308 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e75fa29-31a8-4d98-b568-9f9a7a48a844 None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:57:23.724737 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e75fa29-31a8-4d98-b568-9f9a7a48a844 None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:57:23.725222 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e75fa29-31a8-4d98-b568-9f9a7a48a844 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:57:23.741076 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e75fa29-31a8-4d98-b568-9f9a7a48a844 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 13 15:57:23.747798 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application [None req-0e75fa29-31a8-4d98-b568-9f9a7a48a844 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 13 15:57:23.747798 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:57:23.747798 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:57:23.747798 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:57:23.747798 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:57:23.747798 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:57:23.747798 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:57:23.747798 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:57:23.747798 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:57:23.747798 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:57:23.747798 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:57:23.747798 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:57:23.747798 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 13 15:57:23.747798 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 13 15:57:23.747798 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 13 15:57:23.747798 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 13 15:57:23.749018 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 13 15:57:23.749018 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 13 15:57:23.749018 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 13 15:57:23.749018 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 13 15:57:23.749018 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:57:23.749018 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:57:23.749018 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:57:23.749018 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:57:23.749018 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:57:23.749018 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:57:23.749018 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:57:23.749018 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application with Lock( Sep 13 15:57:23.749018 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:57:23.749018 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:57:23.749018 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:57:23.749018 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:57:23.750560 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:57:23.750560 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:57:23.750560 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:57:23.750560 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:57:23.750560 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 13 15:57:23.750560 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 13 15:57:23.750560 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 13 15:57:23.750560 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 13 15:57:23.750560 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 13 15:57:23.750560 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 13 15:57:23.750560 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 13 15:57:23.750560 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application Sep 13 15:57:23.750560 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 634/1249] 10.222.0.50 () {56 vars in 1096 bytes} [Tue Sep 13 15:57:23 2022] GET /identity/v3/roles/admin => generated 83 bytes in 181 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:57:23.756515 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bfa8a93-e964-4c46-9899-5edfb6c0f209 None None] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:57:23.868980 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bfa8a93-e964-4c46-9899-5edfb6c0f209 None None] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:57:23.898433 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:57:23.906712 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bcd110e-9693-44ee-be56-a39910f5b624 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:57:23.907431 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bcd110e-9693-44ee-be56-a39910f5b624 None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:57:23.907723 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bcd110e-9693-44ee-be56-a39910f5b624 None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:57:23.908004 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bcd110e-9693-44ee-be56-a39910f5b624 None admin] PATH_INFO: `/v3/roles` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:57:23.908679 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8bcd110e-9693-44ee-be56-a39910f5b624 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81438) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:57:23.908985 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8bcd110e-9693-44ee-be56-a39910f5b624 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:57:23.910351 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8bcd110e-9693-44ee-be56-a39910f5b624 None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:57:23.923099 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 616/1250] 10.222.0.50 () {56 vars in 1099 bytes} [Tue Sep 13 15:57:23 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:57:23.929510 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e75fa29-31a8-4d98-b568-9f9a7a48a844 None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:57:24.016394 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:57:24.041989 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e75fa29-31a8-4d98-b568-9f9a7a48a844 None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:57:24.077570 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06a284ed-4198-41ad-8e5f-4918c66bb9e8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:57:24.078751 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06a284ed-4198-41ad-8e5f-4918c66bb9e8 None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:57:24.079014 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06a284ed-4198-41ad-8e5f-4918c66bb9e8 None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:57:24.079288 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06a284ed-4198-41ad-8e5f-4918c66bb9e8 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:57:24.089982 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06a284ed-4198-41ad-8e5f-4918c66bb9e8 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:57:24.091801 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06a284ed-4198-41ad-8e5f-4918c66bb9e8 None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:57:24.108529 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application [None req-06a284ed-4198-41ad-8e5f-4918c66bb9e8 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:57:24.108529 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:57:24.108529 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:57:24.108529 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:57:24.108529 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:57:24.108529 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:57:24.108529 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:57:24.108529 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:57:24.108529 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:57:24.108529 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:57:24.108529 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:57:24.108529 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:57:24.108529 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 13 15:57:24.108529 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 13 15:57:24.108529 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 13 15:57:24.108529 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 13 15:57:24.110244 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:57:24.110244 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:57:24.110244 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:57:24.110244 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:57:24.110244 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:57:24.110244 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:57:24.110244 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:57:24.110244 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application with Lock( Sep 13 15:57:24.110244 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:57:24.110244 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:57:24.110244 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:57:24.110244 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:57:24.110244 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:57:24.110244 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:57:24.110244 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:57:24.110244 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:57:24.110244 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 13 15:57:24.111330 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 13 15:57:24.111330 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:57:24.111330 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application Sep 13 15:57:24.111330 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 635/1251] 10.222.0.50 () {56 vars in 1108 bytes} [Tue Sep 13 15:57:23 2022] GET /identity/v3/domains/Default => generated 87 bytes in 181 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:57:24.115925 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bcd110e-9693-44ee-be56-a39910f5b624 None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:57:24.244111 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bcd110e-9693-44ee-be56-a39910f5b624 None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:57:24.281647 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0570118a-93e6-43ce-9f5f-be28aecc441a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:57:24.282917 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0570118a-93e6-43ce-9f5f-be28aecc441a None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:57:24.283206 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0570118a-93e6-43ce-9f5f-be28aecc441a None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:57:24.283477 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0570118a-93e6-43ce-9f5f-be28aecc441a None admin] PATH_INFO: `/v3/domains` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:57:24.284125 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0570118a-93e6-43ce-9f5f-be28aecc441a None admin] RBAC: Adding query filter params (name=Default) {{(pid=81438) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:57:24.284426 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0570118a-93e6-43ce-9f5f-be28aecc441a None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:57:24.285800 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0570118a-93e6-43ce-9f5f-be28aecc441a None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:57:24.299030 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 617/1252] 10.222.0.50 () {56 vars in 1109 bytes} [Tue Sep 13 15:57:24 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:57:24.305541 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06a284ed-4198-41ad-8e5f-4918c66bb9e8 None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:57:24.411075 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06a284ed-4198-41ad-8e5f-4918c66bb9e8 None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:57:24.438774 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:57:24.447222 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1db51a9c-1d98-4b85-bf5b-dc1f5f175be2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:57:24.447876 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1db51a9c-1d98-4b85-bf5b-dc1f5f175be2 None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:57:24.448155 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1db51a9c-1d98-4b85-bf5b-dc1f5f175be2 None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:57:24.448429 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1db51a9c-1d98-4b85-bf5b-dc1f5f175be2 None admin] PATH_INFO: `/v3/users/placement` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:57:24.457627 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1db51a9c-1d98-4b85-bf5b-dc1f5f175be2 None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:57:24.459107 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1db51a9c-1d98-4b85-bf5b-dc1f5f175be2 None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:57:24.472411 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application [None req-1db51a9c-1d98-4b85-bf5b-dc1f5f175be2 None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Sep 13 15:57:24.472411 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:57:24.472411 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:57:24.472411 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:57:24.472411 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:57:24.472411 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:57:24.472411 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:57:24.472411 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:57:24.472411 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:57:24.472411 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:57:24.472411 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:57:24.472411 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:57:24.472411 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 13 15:57:24.472411 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 13 15:57:24.472411 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 13 15:57:24.472411 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 13 15:57:24.473622 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:57:24.473622 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:57:24.473622 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 13 15:57:24.473622 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 13 15:57:24.473622 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 13 15:57:24.473622 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 13 15:57:24.473622 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:57:24.473622 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:57:24.473622 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:57:24.473622 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:57:24.473622 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:57:24.473622 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application with Lock( Sep 13 15:57:24.473622 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:57:24.473622 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:57:24.473622 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:57:24.473622 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:57:24.473622 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:57:24.473622 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:57:24.475089 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:57:24.475089 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:57:24.475089 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 13 15:57:24.475089 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 13 15:57:24.475089 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 13 15:57:24.475089 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 13 15:57:24.475089 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 13 15:57:24.475089 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 13 15:57:24.475089 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Sep 13 15:57:24.475089 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application Sep 13 15:57:24.475089 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 636/1253] 10.222.0.50 () {56 vars in 1108 bytes} [Tue Sep 13 15:57:24 2022] GET /identity/v3/users/placement => generated 87 bytes in 169 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:57:24.479546 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0570118a-93e6-43ce-9f5f-be28aecc441a None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:57:24.539158 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:57:24.585474 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0570118a-93e6-43ce-9f5f-be28aecc441a None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:57:24.620630 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e68b7a8d-3059-48a1-b525-3051f9e3f032 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:57:24.621237 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e68b7a8d-3059-48a1-b525-3051f9e3f032 None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:57:24.621517 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e68b7a8d-3059-48a1-b525-3051f9e3f032 None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:57:24.621794 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e68b7a8d-3059-48a1-b525-3051f9e3f032 None admin] PATH_INFO: `/v3/users` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:57:24.622502 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e68b7a8d-3059-48a1-b525-3051f9e3f032 None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=81438) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:57:24.622806 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e68b7a8d-3059-48a1-b525-3051f9e3f032 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:57:24.624259 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e68b7a8d-3059-48a1-b525-3051f9e3f032 None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:57:24.642850 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 618/1254] 10.222.0.50 () {56 vars in 1143 bytes} [Tue Sep 13 15:57:24 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:57:24.648655 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1db51a9c-1d98-4b85-bf5b-dc1f5f175be2 None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:57:24.762202 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1db51a9c-1d98-4b85-bf5b-dc1f5f175be2 None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:57:24.800068 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5c5658d-35c1-4c34-b8d9-aa4362283a66 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:57:24.800595 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5c5658d-35c1-4c34-b8d9-aa4362283a66 None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:57:24.800870 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5c5658d-35c1-4c34-b8d9-aa4362283a66 None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:57:24.801403 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5c5658d-35c1-4c34-b8d9-aa4362283a66 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:57:24.857812 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:57:24.870781 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5c5658d-35c1-4c34-b8d9-aa4362283a66 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:57:24.872272 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5c5658d-35c1-4c34-b8d9-aa4362283a66 None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:57:24.967744 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 637/1255] 10.222.0.50 () {58 vars in 1282 bytes} [Tue Sep 13 15:57:24 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 320 msecs (HTTP/1.1 200) 6 headers in 359 bytes (2 switches on core 0) Sep 13 15:57:24.973592 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e68b7a8d-3059-48a1-b525-3051f9e3f032 None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:57:25.072335 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e68b7a8d-3059-48a1-b525-3051f9e3f032 None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:57:25.085394 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:57:25.106052 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8f1141b-b2d7-4e6e-adc1-0a61dcf1faf8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:57:25.106676 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8f1141b-b2d7-4e6e-adc1-0a61dcf1faf8 None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:57:25.106990 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8f1141b-b2d7-4e6e-adc1-0a61dcf1faf8 None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:57:25.107266 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8f1141b-b2d7-4e6e-adc1-0a61dcf1faf8 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:57:25.117691 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8f1141b-b2d7-4e6e-adc1-0a61dcf1faf8 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:57:25.119159 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8f1141b-b2d7-4e6e-adc1-0a61dcf1faf8 None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:57:25.133848 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application [None req-b8f1141b-b2d7-4e6e-adc1-0a61dcf1faf8 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:57:25.133848 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:57:25.133848 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:57:25.133848 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:57:25.133848 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:57:25.133848 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:57:25.133848 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:57:25.133848 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:57:25.133848 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:57:25.133848 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:57:25.133848 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:57:25.133848 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:57:25.133848 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 13 15:57:25.133848 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 13 15:57:25.133848 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 13 15:57:25.133848 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 13 15:57:25.135116 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:57:25.135116 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:57:25.135116 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:57:25.135116 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:57:25.135116 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:57:25.135116 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:57:25.135116 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:57:25.135116 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application with Lock( Sep 13 15:57:25.135116 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:57:25.135116 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:57:25.135116 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:57:25.135116 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:57:25.135116 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:57:25.135116 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:57:25.135116 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:57:25.135116 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:57:25.135116 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 13 15:57:25.136239 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 13 15:57:25.136239 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:57:25.136239 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Sep 13 15:57:25.136239 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 619/1256] 10.222.0.50 () {56 vars in 1108 bytes} [Tue Sep 13 15:57:24 2022] GET /identity/v3/domains/Default => generated 87 bytes in 162 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:57:25.140631 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5c5658d-35c1-4c34-b8d9-aa4362283a66 None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:57:25.224091 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:57:25.237052 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5c5658d-35c1-4c34-b8d9-aa4362283a66 None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:57:25.269969 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf6806be-823c-433e-9f2b-27ee8240981e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:57:25.270557 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf6806be-823c-433e-9f2b-27ee8240981e None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:57:25.270834 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf6806be-823c-433e-9f2b-27ee8240981e None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:57:25.271134 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf6806be-823c-433e-9f2b-27ee8240981e None admin] PATH_INFO: `/v3/domains` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:57:25.271766 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf6806be-823c-433e-9f2b-27ee8240981e None admin] RBAC: Adding query filter params (name=Default) {{(pid=81437) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:57:25.272064 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf6806be-823c-433e-9f2b-27ee8240981e None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:57:25.273487 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf6806be-823c-433e-9f2b-27ee8240981e None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:57:25.284808 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 638/1257] 10.222.0.50 () {56 vars in 1109 bytes} [Tue Sep 13 15:57:25 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 145 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:57:25.290471 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8f1141b-b2d7-4e6e-adc1-0a61dcf1faf8 None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:57:25.386737 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8f1141b-b2d7-4e6e-adc1-0a61dcf1faf8 None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:57:25.418595 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c47feb13-26cb-4fc6-aff5-0379520fd1f1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:57:25.419230 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c47feb13-26cb-4fc6-aff5-0379520fd1f1 None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:57:25.419506 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c47feb13-26cb-4fc6-aff5-0379520fd1f1 None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:57:25.419775 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c47feb13-26cb-4fc6-aff5-0379520fd1f1 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:57:25.427021 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c47feb13-26cb-4fc6-aff5-0379520fd1f1 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:57:25.428440 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c47feb13-26cb-4fc6-aff5-0379520fd1f1 None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:57:25.439507 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application [None req-c47feb13-26cb-4fc6-aff5-0379520fd1f1 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 13 15:57:25.439507 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:57:25.439507 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:57:25.439507 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:57:25.439507 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:57:25.439507 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:57:25.439507 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:57:25.439507 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:57:25.439507 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:57:25.439507 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:57:25.439507 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:57:25.439507 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:57:25.439507 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 13 15:57:25.439507 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 13 15:57:25.439507 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 13 15:57:25.440550 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 13 15:57:25.440550 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:57:25.440550 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:57:25.440550 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:57:25.440550 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:57:25.440550 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:57:25.440550 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:57:25.440550 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:57:25.440550 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application with Lock( Sep 13 15:57:25.440550 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:57:25.440550 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:57:25.440550 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:57:25.440550 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:57:25.440550 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:57:25.440550 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:57:25.440550 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:57:25.440550 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:57:25.441667 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 13 15:57:25.441667 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 13 15:57:25.441667 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 13 15:57:25.441667 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 13 15:57:25.441667 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 13 15:57:25.441667 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 13 15:57:25.441667 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 13 15:57:25.441667 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Sep 13 15:57:25.441667 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 620/1258] 10.222.0.50 () {56 vars in 1111 bytes} [Tue Sep 13 15:57:25 2022] GET /identity/v3/projects/service => generated 88 bytes in 151 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:57:25.446273 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf6806be-823c-433e-9f2b-27ee8240981e None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:57:25.548049 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf6806be-823c-433e-9f2b-27ee8240981e None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:57:25.580517 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d676f376-1eba-4204-aed2-50eb3a6c2606 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:57:25.581103 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d676f376-1eba-4204-aed2-50eb3a6c2606 None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:57:25.581377 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d676f376-1eba-4204-aed2-50eb3a6c2606 None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:57:25.581643 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d676f376-1eba-4204-aed2-50eb3a6c2606 None admin] PATH_INFO: `/v3/projects` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:57:25.582311 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d676f376-1eba-4204-aed2-50eb3a6c2606 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81437) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:57:25.582608 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d676f376-1eba-4204-aed2-50eb3a6c2606 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:57:25.584050 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d676f376-1eba-4204-aed2-50eb3a6c2606 None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:57:25.595975 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 639/1259] 10.222.0.50 () {56 vars in 1148 bytes} [Tue Sep 13 15:57:25 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:57:25.601977 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c47feb13-26cb-4fc6-aff5-0379520fd1f1 None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:57:25.627137 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:57:25.698893 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c47feb13-26cb-4fc6-aff5-0379520fd1f1 None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:57:25.734925 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9206edb-2051-4f03-b503-694e75e9e94e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:57:25.735729 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9206edb-2051-4f03-b503-694e75e9e94e None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:57:25.736012 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9206edb-2051-4f03-b503-694e75e9e94e None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:57:25.736308 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9206edb-2051-4f03-b503-694e75e9e94e None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:57:25.737156 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9206edb-2051-4f03-b503-694e75e9e94e None admin] RBAC: Adding query filter params (role.id=031d4e1a23e34f71858fc4a6dfbd4ccb, scope.project.id=5811e3b83a1e462e9320cad79a3ad791, user.id=31480b61418340bd88583f034f67412e) {{(pid=81438) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:57:25.737499 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9206edb-2051-4f03-b503-694e75e9e94e None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:57:25.739319 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9206edb-2051-4f03-b503-694e75e9e94e None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:57:25.745701 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 621/1260] 10.222.0.50 () {56 vars in 1374 bytes} [Tue Sep 13 15:57:25 2022] GET /identity/v3/role_assignments?user.id=31480b61418340bd88583f034f67412e&scope.project.id=5811e3b83a1e462e9320cad79a3ad791&role.id=031d4e1a23e34f71858fc4a6dfbd4ccb => generated 259 bytes in 144 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:57:27.866696 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c7c0cc5-aa45-49f3-85d5-71a50a477682 None None] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:57:27.867272 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c7c0cc5-aa45-49f3-85d5-71a50a477682 None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:57:27.867272 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c7c0cc5-aa45-49f3-85d5-71a50a477682 None None] PATH_INFO: `/` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:57:27.868607 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 640/1261] 10.222.0.50 () {54 vars in 927 bytes} [Tue Sep 13 15:57:27 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:57:27.873726 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b6dd909-33dd-4a6f-ac11-eeb2bde9a9f9 None None] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:57:27.874138 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b6dd909-33dd-4a6f-ac11-eeb2bde9a9f9 None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:57:27.874276 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b6dd909-33dd-4a6f-ac11-eeb2bde9a9f9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:57:27.892608 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.auth.core [None req-4b6dd909-33dd-4a6f-ac11-eeb2bde9a9f9 None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81438) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:57:27.893775 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-4b6dd909-33dd-4a6f-ac11-eeb2bde9a9f9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:57:27.957603 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 622/1262] 10.222.0.50 () {58 vars in 1024 bytes} [Tue Sep 13 15:57:27 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 13 15:57:28.227061 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c7c0cc5-aa45-49f3-85d5-71a50a477682 None None] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:57:28.237929 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:57:28.325352 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c7c0cc5-aa45-49f3-85d5-71a50a477682 None None] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:57:28.357683 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f6dbf6e-c500-43a2-9b53-de8c96de8a71 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:57:28.358293 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f6dbf6e-c500-43a2-9b53-de8c96de8a71 None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:57:28.358568 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f6dbf6e-c500-43a2-9b53-de8c96de8a71 None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:57:28.358863 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f6dbf6e-c500-43a2-9b53-de8c96de8a71 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:57:28.368533 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f6dbf6e-c500-43a2-9b53-de8c96de8a71 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 13 15:57:28.373178 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application [None req-1f6dbf6e-c500-43a2-9b53-de8c96de8a71 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 13 15:57:28.373178 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:57:28.373178 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:57:28.373178 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:57:28.373178 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:57:28.373178 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:57:28.373178 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:57:28.373178 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:57:28.373178 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:57:28.373178 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:57:28.373178 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:57:28.373178 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:57:28.373178 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 13 15:57:28.373178 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 13 15:57:28.373178 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 13 15:57:28.373178 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 13 15:57:28.374278 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 13 15:57:28.374278 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 13 15:57:28.374278 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 13 15:57:28.374278 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 13 15:57:28.374278 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:57:28.374278 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:57:28.374278 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:57:28.374278 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:57:28.374278 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:57:28.374278 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:57:28.374278 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:57:28.374278 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application with Lock( Sep 13 15:57:28.374278 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:57:28.374278 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:57:28.374278 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:57:28.374278 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:57:28.375840 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:57:28.375840 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:57:28.375840 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:57:28.375840 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:57:28.375840 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 13 15:57:28.375840 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 13 15:57:28.375840 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 13 15:57:28.375840 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 13 15:57:28.375840 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 13 15:57:28.375840 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 13 15:57:28.375840 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 13 15:57:28.375840 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application Sep 13 15:57:28.375840 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 641/1263] 10.222.0.50 () {56 vars in 1096 bytes} [Tue Sep 13 15:57:28 2022] GET /identity/v3/roles/admin => generated 83 bytes in 148 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:57:28.380757 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b6dd909-33dd-4a6f-ac11-eeb2bde9a9f9 None None] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:57:28.399468 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:57:28.480110 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b6dd909-33dd-4a6f-ac11-eeb2bde9a9f9 None None] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:57:28.512946 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-360cae65-f3bf-46c1-9805-ec64430a8ff5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:57:28.513547 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-360cae65-f3bf-46c1-9805-ec64430a8ff5 None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:57:28.513831 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-360cae65-f3bf-46c1-9805-ec64430a8ff5 None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:57:28.514107 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-360cae65-f3bf-46c1-9805-ec64430a8ff5 None admin] PATH_INFO: `/v3/roles` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:57:28.514760 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-360cae65-f3bf-46c1-9805-ec64430a8ff5 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81438) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:57:28.515749 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-360cae65-f3bf-46c1-9805-ec64430a8ff5 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:57:28.517231 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-360cae65-f3bf-46c1-9805-ec64430a8ff5 None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:57:28.525051 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 623/1264] 10.222.0.50 () {56 vars in 1099 bytes} [Tue Sep 13 15:57:28 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 145 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:57:28.530696 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f6dbf6e-c500-43a2-9b53-de8c96de8a71 None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:57:28.596465 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:57:28.630124 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f6dbf6e-c500-43a2-9b53-de8c96de8a71 None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:57:28.663453 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb249445-0975-42c1-a430-6aa6aad3f4f9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:57:28.664066 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb249445-0975-42c1-a430-6aa6aad3f4f9 None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:57:28.664346 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb249445-0975-42c1-a430-6aa6aad3f4f9 None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:57:28.664618 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb249445-0975-42c1-a430-6aa6aad3f4f9 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:57:28.674726 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb249445-0975-42c1-a430-6aa6aad3f4f9 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:57:28.676473 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb249445-0975-42c1-a430-6aa6aad3f4f9 None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:57:28.691051 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application [None req-cb249445-0975-42c1-a430-6aa6aad3f4f9 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:57:28.691051 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:57:28.691051 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:57:28.691051 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:57:28.691051 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:57:28.691051 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:57:28.691051 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:57:28.691051 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:57:28.691051 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:57:28.691051 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:57:28.691051 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:57:28.691051 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:57:28.691051 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 13 15:57:28.691051 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 13 15:57:28.691051 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 13 15:57:28.691051 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 13 15:57:28.692192 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:57:28.692192 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:57:28.692192 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:57:28.692192 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:57:28.692192 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:57:28.692192 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:57:28.692192 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:57:28.692192 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application with Lock( Sep 13 15:57:28.692192 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:57:28.692192 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:57:28.692192 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:57:28.692192 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:57:28.692192 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:57:28.692192 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:57:28.692192 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:57:28.692192 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:57:28.692192 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 13 15:57:28.693383 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 13 15:57:28.693383 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:57:28.693383 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application Sep 13 15:57:28.693383 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 642/1265] 10.222.0.50 () {56 vars in 1108 bytes} [Tue Sep 13 15:57:28 2022] GET /identity/v3/domains/Default => generated 87 bytes in 162 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:57:28.697749 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-360cae65-f3bf-46c1-9805-ec64430a8ff5 None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:57:28.791457 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:57:28.812953 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-360cae65-f3bf-46c1-9805-ec64430a8ff5 None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:57:28.844833 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efc9df75-5ef2-45e4-9a5c-dd5387bb6796 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:57:28.845439 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efc9df75-5ef2-45e4-9a5c-dd5387bb6796 None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:57:28.845723 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efc9df75-5ef2-45e4-9a5c-dd5387bb6796 None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:57:28.846001 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efc9df75-5ef2-45e4-9a5c-dd5387bb6796 None admin] PATH_INFO: `/v3/domains` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:57:28.846639 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efc9df75-5ef2-45e4-9a5c-dd5387bb6796 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81438) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:57:28.846973 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efc9df75-5ef2-45e4-9a5c-dd5387bb6796 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:57:28.848341 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efc9df75-5ef2-45e4-9a5c-dd5387bb6796 None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:57:28.860214 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 624/1266] 10.222.0.50 () {56 vars in 1109 bytes} [Tue Sep 13 15:57:28 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:57:28.865907 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb249445-0975-42c1-a430-6aa6aad3f4f9 None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:57:28.962595 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb249445-0975-42c1-a430-6aa6aad3f4f9 None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:57:28.981472 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:57:28.995614 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1c662db-314e-43ee-906b-2a061a702587 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:57:28.996222 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1c662db-314e-43ee-906b-2a061a702587 None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:57:28.996499 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1c662db-314e-43ee-906b-2a061a702587 None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:57:28.996767 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1c662db-314e-43ee-906b-2a061a702587 None admin] PATH_INFO: `/v3/users/placement` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:57:29.003935 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1c662db-314e-43ee-906b-2a061a702587 None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:57:29.005357 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1c662db-314e-43ee-906b-2a061a702587 None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:57:29.016106 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application [None req-d1c662db-314e-43ee-906b-2a061a702587 None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Sep 13 15:57:29.016106 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:57:29.016106 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:57:29.016106 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:57:29.016106 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:57:29.016106 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:57:29.016106 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:57:29.016106 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:57:29.016106 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:57:29.016106 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:57:29.016106 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:57:29.016106 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:57:29.016106 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 13 15:57:29.016106 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 13 15:57:29.016106 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 13 15:57:29.016106 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 13 15:57:29.017223 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:57:29.017223 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:57:29.017223 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 13 15:57:29.017223 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 13 15:57:29.017223 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 13 15:57:29.017223 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 13 15:57:29.017223 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:57:29.017223 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:57:29.017223 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:57:29.017223 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:57:29.017223 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:57:29.017223 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application with Lock( Sep 13 15:57:29.017223 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:57:29.017223 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:57:29.017223 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:57:29.017223 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:57:29.017223 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:57:29.017223 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:57:29.018415 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:57:29.018415 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:57:29.018415 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 13 15:57:29.018415 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 13 15:57:29.018415 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 13 15:57:29.018415 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 13 15:57:29.018415 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 13 15:57:29.018415 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 13 15:57:29.018415 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Sep 13 15:57:29.018415 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application Sep 13 15:57:29.018415 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 643/1267] 10.222.0.50 () {56 vars in 1108 bytes} [Tue Sep 13 15:57:28 2022] GET /identity/v3/users/placement => generated 87 bytes in 152 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:57:29.022951 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efc9df75-5ef2-45e4-9a5c-dd5387bb6796 None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:57:29.121258 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efc9df75-5ef2-45e4-9a5c-dd5387bb6796 None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:57:29.153249 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b349143c-82ba-48f4-ac82-0d2b3c44d757 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:57:29.153841 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b349143c-82ba-48f4-ac82-0d2b3c44d757 None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:57:29.154118 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b349143c-82ba-48f4-ac82-0d2b3c44d757 None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:57:29.154389 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b349143c-82ba-48f4-ac82-0d2b3c44d757 None admin] PATH_INFO: `/v3/users` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:57:29.155126 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b349143c-82ba-48f4-ac82-0d2b3c44d757 None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=81438) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:57:29.155427 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b349143c-82ba-48f4-ac82-0d2b3c44d757 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:57:29.157350 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b349143c-82ba-48f4-ac82-0d2b3c44d757 None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:57:29.165549 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:57:29.174404 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 625/1268] 10.222.0.50 () {56 vars in 1143 bytes} [Tue Sep 13 15:57:29 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:57:29.180501 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1c662db-314e-43ee-906b-2a061a702587 None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:57:29.276743 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1c662db-314e-43ee-906b-2a061a702587 None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:57:29.309714 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d927a6dd-1ed4-4a79-9830-c15b553e67fa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:57:29.310322 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d927a6dd-1ed4-4a79-9830-c15b553e67fa None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:57:29.310597 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d927a6dd-1ed4-4a79-9830-c15b553e67fa None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:57:29.310869 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d927a6dd-1ed4-4a79-9830-c15b553e67fa None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:57:29.321041 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d927a6dd-1ed4-4a79-9830-c15b553e67fa None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:57:29.322411 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d927a6dd-1ed4-4a79-9830-c15b553e67fa None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:57:29.336522 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application [None req-d927a6dd-1ed4-4a79-9830-c15b553e67fa None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:57:29.336522 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:57:29.336522 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:57:29.336522 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:57:29.336522 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:57:29.336522 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:57:29.336522 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:57:29.336522 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:57:29.336522 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:57:29.336522 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:57:29.336522 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:57:29.336522 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:57:29.336522 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 13 15:57:29.336522 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 13 15:57:29.336522 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 13 15:57:29.336522 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 13 15:57:29.337645 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:57:29.337645 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:57:29.337645 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:57:29.337645 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:57:29.337645 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:57:29.337645 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:57:29.337645 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:57:29.337645 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application with Lock( Sep 13 15:57:29.337645 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:57:29.337645 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:57:29.337645 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:57:29.337645 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:57:29.337645 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:57:29.337645 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:57:29.337645 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:57:29.337645 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:57:29.337645 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 13 15:57:29.338789 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 13 15:57:29.338789 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:57:29.338789 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application Sep 13 15:57:29.338789 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 644/1269] 10.222.0.50 () {56 vars in 1108 bytes} [Tue Sep 13 15:57:29 2022] GET /identity/v3/domains/Default => generated 87 bytes in 158 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:57:29.344043 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b349143c-82ba-48f4-ac82-0d2b3c44d757 None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:57:29.439909 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b349143c-82ba-48f4-ac82-0d2b3c44d757 None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:57:29.473157 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa85939c-e10b-4da6-8826-bb65ffd6d0fb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:57:29.473749 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa85939c-e10b-4da6-8826-bb65ffd6d0fb None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:57:29.474028 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa85939c-e10b-4da6-8826-bb65ffd6d0fb None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:57:29.474303 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa85939c-e10b-4da6-8826-bb65ffd6d0fb None admin] PATH_INFO: `/v3/domains` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:57:29.474955 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa85939c-e10b-4da6-8826-bb65ffd6d0fb None admin] RBAC: Adding query filter params (name=Default) {{(pid=81438) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:57:29.475260 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa85939c-e10b-4da6-8826-bb65ffd6d0fb None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:57:29.476614 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa85939c-e10b-4da6-8826-bb65ffd6d0fb None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:57:29.487416 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 626/1270] 10.222.0.50 () {56 vars in 1109 bytes} [Tue Sep 13 15:57:29 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 144 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:57:29.493010 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d927a6dd-1ed4-4a79-9830-c15b553e67fa None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:57:29.524503 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:57:29.591387 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d927a6dd-1ed4-4a79-9830-c15b553e67fa None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:57:29.624447 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea6037ad-c049-480d-8c27-14ecb953c8d7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:57:29.625064 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea6037ad-c049-480d-8c27-14ecb953c8d7 None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:57:29.625344 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea6037ad-c049-480d-8c27-14ecb953c8d7 None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:57:29.625619 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea6037ad-c049-480d-8c27-14ecb953c8d7 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:57:29.632076 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea6037ad-c049-480d-8c27-14ecb953c8d7 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:57:29.633522 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea6037ad-c049-480d-8c27-14ecb953c8d7 None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:57:29.644920 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application [None req-ea6037ad-c049-480d-8c27-14ecb953c8d7 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 13 15:57:29.644920 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:57:29.644920 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:57:29.644920 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:57:29.644920 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:57:29.644920 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:57:29.644920 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:57:29.644920 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:57:29.644920 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:57:29.644920 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:57:29.644920 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:57:29.644920 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:57:29.644920 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 13 15:57:29.644920 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 13 15:57:29.644920 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 13 15:57:29.645968 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 13 15:57:29.645968 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:57:29.645968 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:57:29.645968 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:57:29.645968 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:57:29.645968 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:57:29.645968 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:57:29.645968 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:57:29.645968 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application with Lock( Sep 13 15:57:29.645968 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:57:29.645968 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:57:29.645968 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:57:29.645968 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:57:29.645968 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:57:29.645968 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:57:29.645968 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:57:29.645968 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:57:29.651693 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 13 15:57:29.651693 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 13 15:57:29.651693 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 13 15:57:29.651693 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 13 15:57:29.651693 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 13 15:57:29.651693 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 13 15:57:29.651693 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 13 15:57:29.651693 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application Sep 13 15:57:29.651693 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 645/1271] 10.222.0.50 () {56 vars in 1111 bytes} [Tue Sep 13 15:57:29 2022] GET /identity/v3/projects/service => generated 88 bytes in 153 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:57:29.652323 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa85939c-e10b-4da6-8826-bb65ffd6d0fb None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:57:29.708625 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:57:29.754672 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa85939c-e10b-4da6-8826-bb65ffd6d0fb None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:57:29.790711 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d73b41b0-09d0-42c6-b51f-d5dfd0833a19 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:57:29.791339 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d73b41b0-09d0-42c6-b51f-d5dfd0833a19 None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:57:29.791621 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d73b41b0-09d0-42c6-b51f-d5dfd0833a19 None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:57:29.791896 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d73b41b0-09d0-42c6-b51f-d5dfd0833a19 None admin] PATH_INFO: `/v3/projects` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:57:29.792575 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d73b41b0-09d0-42c6-b51f-d5dfd0833a19 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81438) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:57:29.792881 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d73b41b0-09d0-42c6-b51f-d5dfd0833a19 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:57:29.794303 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d73b41b0-09d0-42c6-b51f-d5dfd0833a19 None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:57:29.806415 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 627/1272] 10.222.0.50 () {56 vars in 1148 bytes} [Tue Sep 13 15:57:29 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:57:29.813130 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea6037ad-c049-480d-8c27-14ecb953c8d7 None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:57:29.900921 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:57:29.909662 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea6037ad-c049-480d-8c27-14ecb953c8d7 None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:57:29.942076 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c747976e-d8bd-42ac-830c-f41a23844c79 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:57:29.942710 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c747976e-d8bd-42ac-830c-f41a23844c79 None admin] REQUEST_METHOD: `PUT` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:57:29.943014 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c747976e-d8bd-42ac-830c-f41a23844c79 None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:57:29.943288 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c747976e-d8bd-42ac-830c-f41a23844c79 None admin] PATH_INFO: `/v3/projects/5811e3b83a1e462e9320cad79a3ad791/users/31480b61418340bd88583f034f67412e/roles/031d4e1a23e34f71858fc4a6dfbd4ccb` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:57:29.945121 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c747976e-d8bd-42ac-830c-f41a23844c79 None admin] RBAC: Authorizing `identity:create_grant(project_id=5811e3b83a1e462e9320cad79a3ad791, user_id=31480b61418340bd88583f034f67412e, role_id=031d4e1a23e34f71858fc4a6dfbd4ccb)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:57:29.946536 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c747976e-d8bd-42ac-830c-f41a23844c79 None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:57:29.959047 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 646/1273] 10.222.0.50 () {58 vars in 1439 bytes} [Tue Sep 13 15:57:29 2022] PUT /identity/v3/projects/5811e3b83a1e462e9320cad79a3ad791/users/31480b61418340bd88583f034f67412e/roles/031d4e1a23e34f71858fc4a6dfbd4ccb => generated 0 bytes in 147 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 13 15:57:32.075281 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a3f0877-369d-468c-a2e7-805f9f6d721c None None] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:57:32.075699 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a3f0877-369d-468c-a2e7-805f9f6d721c None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:57:32.075840 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a3f0877-369d-468c-a2e7-805f9f6d721c None None] PATH_INFO: `/` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:57:32.077844 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 628/1274] 10.222.0.50 () {54 vars in 927 bytes} [Tue Sep 13 15:57:32 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:57:32.082955 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1a83f49-04e4-4091-98b3-31ec27319e03 None None] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:57:32.083404 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1a83f49-04e4-4091-98b3-31ec27319e03 None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:57:32.083500 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1a83f49-04e4-4091-98b3-31ec27319e03 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:57:32.101368 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.auth.core [None req-d1a83f49-04e4-4091-98b3-31ec27319e03 None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81437) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:57:32.102569 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-d1a83f49-04e4-4091-98b3-31ec27319e03 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:57:32.166771 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 647/1275] 10.222.0.50 () {58 vars in 1024 bytes} [Tue Sep 13 15:57:32 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 13 15:57:32.433586 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a3f0877-369d-468c-a2e7-805f9f6d721c None None] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:57:32.532603 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a3f0877-369d-468c-a2e7-805f9f6d721c None None] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:57:32.550889 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:57:32.568149 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-839801bd-5e80-4bad-b44e-503dc958c08f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:57:32.568766 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-839801bd-5e80-4bad-b44e-503dc958c08f None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:57:32.569048 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-839801bd-5e80-4bad-b44e-503dc958c08f None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:57:32.569324 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-839801bd-5e80-4bad-b44e-503dc958c08f None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:57:32.579190 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-839801bd-5e80-4bad-b44e-503dc958c08f None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 13 15:57:32.583917 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application [None req-839801bd-5e80-4bad-b44e-503dc958c08f None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 13 15:57:32.583917 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:57:32.583917 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:57:32.583917 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:57:32.583917 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:57:32.583917 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:57:32.583917 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:57:32.583917 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:57:32.583917 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:57:32.583917 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:57:32.583917 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:57:32.583917 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:57:32.583917 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 13 15:57:32.583917 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 13 15:57:32.583917 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 13 15:57:32.583917 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 13 15:57:32.585101 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 13 15:57:32.585101 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 13 15:57:32.585101 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 13 15:57:32.585101 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 13 15:57:32.585101 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:57:32.585101 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:57:32.585101 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:57:32.585101 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:57:32.585101 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:57:32.585101 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:57:32.585101 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:57:32.585101 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application with Lock( Sep 13 15:57:32.585101 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:57:32.585101 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:57:32.585101 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:57:32.585101 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:57:32.586161 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:57:32.586161 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:57:32.586161 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:57:32.586161 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:57:32.586161 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 13 15:57:32.586161 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 13 15:57:32.586161 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 13 15:57:32.586161 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 13 15:57:32.586161 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 13 15:57:32.586161 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 13 15:57:32.586161 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 13 15:57:32.586161 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Sep 13 15:57:32.586161 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 629/1276] 10.222.0.50 () {56 vars in 1096 bytes} [Tue Sep 13 15:57:32 2022] GET /identity/v3/roles/admin => generated 83 bytes in 152 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:57:32.593385 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1a83f49-04e4-4091-98b3-31ec27319e03 None None] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:57:32.671657 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:57:32.691173 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1a83f49-04e4-4091-98b3-31ec27319e03 None None] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:57:32.726060 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2ccc831-acdd-42cd-b811-4f1b66764ba2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:57:32.726764 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2ccc831-acdd-42cd-b811-4f1b66764ba2 None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:57:32.727086 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2ccc831-acdd-42cd-b811-4f1b66764ba2 None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:57:32.727351 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2ccc831-acdd-42cd-b811-4f1b66764ba2 None admin] PATH_INFO: `/v3/roles` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:57:32.728017 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2ccc831-acdd-42cd-b811-4f1b66764ba2 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81437) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:57:32.728391 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2ccc831-acdd-42cd-b811-4f1b66764ba2 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:57:32.729837 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2ccc831-acdd-42cd-b811-4f1b66764ba2 None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:57:32.740291 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 648/1277] 10.222.0.50 () {56 vars in 1099 bytes} [Tue Sep 13 15:57:32 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:57:32.746339 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-839801bd-5e80-4bad-b44e-503dc958c08f None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:57:32.854061 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-839801bd-5e80-4bad-b44e-503dc958c08f None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:57:32.886523 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be33c97f-2e2b-4d55-9353-c825d66107d5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:57:32.887176 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be33c97f-2e2b-4d55-9353-c825d66107d5 None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:57:32.887458 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be33c97f-2e2b-4d55-9353-c825d66107d5 None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:57:32.887730 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be33c97f-2e2b-4d55-9353-c825d66107d5 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:57:32.898091 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be33c97f-2e2b-4d55-9353-c825d66107d5 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:57:32.899598 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be33c97f-2e2b-4d55-9353-c825d66107d5 None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:57:32.913753 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application [None req-be33c97f-2e2b-4d55-9353-c825d66107d5 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:57:32.913753 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:57:32.913753 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:57:32.913753 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:57:32.913753 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:57:32.913753 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:57:32.913753 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:57:32.913753 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:57:32.913753 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:57:32.913753 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:57:32.913753 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:57:32.913753 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:57:32.913753 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 13 15:57:32.913753 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 13 15:57:32.913753 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 13 15:57:32.913753 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 13 15:57:32.914892 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:57:32.914892 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:57:32.914892 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:57:32.914892 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:57:32.914892 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:57:32.914892 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:57:32.914892 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:57:32.914892 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application with Lock( Sep 13 15:57:32.914892 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:57:32.914892 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:57:32.914892 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:57:32.914892 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:57:32.914892 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:57:32.914892 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:57:32.914892 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:57:32.914892 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:57:32.914892 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 13 15:57:32.916227 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 13 15:57:32.916227 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:57:32.916227 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Sep 13 15:57:32.916227 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 630/1278] 10.222.0.50 () {56 vars in 1108 bytes} [Tue Sep 13 15:57:32 2022] GET /identity/v3/domains/Default => generated 87 bytes in 169 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:57:32.920628 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2ccc831-acdd-42cd-b811-4f1b66764ba2 None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:57:33.016164 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2ccc831-acdd-42cd-b811-4f1b66764ba2 None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:57:33.050419 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c1ede0a-185c-4d06-bc49-f44361d2cb15 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:57:33.051300 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c1ede0a-185c-4d06-bc49-f44361d2cb15 None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:57:33.051574 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c1ede0a-185c-4d06-bc49-f44361d2cb15 None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:57:33.051835 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c1ede0a-185c-4d06-bc49-f44361d2cb15 None admin] PATH_INFO: `/v3/domains` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:57:33.052458 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c1ede0a-185c-4d06-bc49-f44361d2cb15 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81437) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:57:33.052753 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c1ede0a-185c-4d06-bc49-f44361d2cb15 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:57:33.054098 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c1ede0a-185c-4d06-bc49-f44361d2cb15 None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:57:33.066021 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 649/1279] 10.222.0.50 () {56 vars in 1109 bytes} [Tue Sep 13 15:57:32 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:57:33.071621 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be33c97f-2e2b-4d55-9353-c825d66107d5 None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:57:33.102530 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:57:33.170315 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be33c97f-2e2b-4d55-9353-c825d66107d5 None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:57:33.204347 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e0139f9-6e2f-4f6d-ba66-06da3595899f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:57:33.204967 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e0139f9-6e2f-4f6d-ba66-06da3595899f None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:57:33.205247 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e0139f9-6e2f-4f6d-ba66-06da3595899f None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:57:33.205522 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e0139f9-6e2f-4f6d-ba66-06da3595899f None admin] PATH_INFO: `/v3/users/placement` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:57:33.212675 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e0139f9-6e2f-4f6d-ba66-06da3595899f None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:57:33.214092 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e0139f9-6e2f-4f6d-ba66-06da3595899f None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:57:33.225513 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application [None req-5e0139f9-6e2f-4f6d-ba66-06da3595899f None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Sep 13 15:57:33.225513 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:57:33.225513 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:57:33.225513 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:57:33.225513 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:57:33.225513 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:57:33.225513 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:57:33.225513 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:57:33.225513 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:57:33.225513 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:57:33.225513 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:57:33.225513 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:57:33.225513 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 13 15:57:33.225513 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 13 15:57:33.225513 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 13 15:57:33.225513 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 13 15:57:33.226644 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:57:33.226644 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:57:33.226644 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 13 15:57:33.226644 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 13 15:57:33.226644 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 13 15:57:33.226644 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 13 15:57:33.226644 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:57:33.226644 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:57:33.226644 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:57:33.226644 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:57:33.226644 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:57:33.226644 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application with Lock( Sep 13 15:57:33.226644 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:57:33.226644 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:57:33.226644 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:57:33.226644 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:57:33.226644 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:57:33.226644 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:57:33.227876 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:57:33.227876 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:57:33.227876 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 13 15:57:33.227876 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 13 15:57:33.227876 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 13 15:57:33.227876 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 13 15:57:33.227876 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 13 15:57:33.227876 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 13 15:57:33.227876 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Sep 13 15:57:33.227876 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Sep 13 15:57:33.227876 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 631/1280] 10.222.0.50 () {56 vars in 1108 bytes} [Tue Sep 13 15:57:33 2022] GET /identity/v3/users/placement => generated 87 bytes in 156 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:57:33.232356 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c1ede0a-185c-4d06-bc49-f44361d2cb15 None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:57:33.236326 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:57:33.330485 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c1ede0a-185c-4d06-bc49-f44361d2cb15 None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:57:33.363677 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0acd7a4d-e3f4-4b5f-9a3e-14e0fa128b22 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:57:33.364270 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0acd7a4d-e3f4-4b5f-9a3e-14e0fa128b22 None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:57:33.364551 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0acd7a4d-e3f4-4b5f-9a3e-14e0fa128b22 None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:57:33.364827 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0acd7a4d-e3f4-4b5f-9a3e-14e0fa128b22 None admin] PATH_INFO: `/v3/users` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:57:33.365542 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0acd7a4d-e3f4-4b5f-9a3e-14e0fa128b22 None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=81437) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:57:33.365851 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0acd7a4d-e3f4-4b5f-9a3e-14e0fa128b22 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:57:33.367317 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0acd7a4d-e3f4-4b5f-9a3e-14e0fa128b22 None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:57:33.383836 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 650/1281] 10.222.0.50 () {56 vars in 1143 bytes} [Tue Sep 13 15:57:33 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:57:33.390019 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e0139f9-6e2f-4f6d-ba66-06da3595899f None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:57:33.477751 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:57:33.487756 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e0139f9-6e2f-4f6d-ba66-06da3595899f None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:57:33.520348 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab36b840-69f9-45a0-afb3-f5324729146c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:57:33.520945 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab36b840-69f9-45a0-afb3-f5324729146c None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:57:33.521222 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab36b840-69f9-45a0-afb3-f5324729146c None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:57:33.521496 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab36b840-69f9-45a0-afb3-f5324729146c None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:57:33.586955 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab36b840-69f9-45a0-afb3-f5324729146c None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:57:33.588318 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab36b840-69f9-45a0-afb3-f5324729146c None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:57:33.672388 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:57:33.682485 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 632/1282] 10.222.0.50 () {58 vars in 1282 bytes} [Tue Sep 13 15:57:33 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 293 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 13 15:57:33.689226 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0acd7a4d-e3f4-4b5f-9a3e-14e0fa128b22 None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:57:33.750716 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:57:33.788828 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0acd7a4d-e3f4-4b5f-9a3e-14e0fa128b22 None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:57:33.822787 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9daa52be-304c-4b10-ab74-08674b55a11d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:57:33.823424 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9daa52be-304c-4b10-ab74-08674b55a11d None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:57:33.823699 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9daa52be-304c-4b10-ab74-08674b55a11d None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:57:33.823966 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9daa52be-304c-4b10-ab74-08674b55a11d None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:57:33.834339 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9daa52be-304c-4b10-ab74-08674b55a11d None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:57:33.835810 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9daa52be-304c-4b10-ab74-08674b55a11d None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:57:33.850753 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application [None req-9daa52be-304c-4b10-ab74-08674b55a11d None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:57:33.850753 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:57:33.850753 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:57:33.850753 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:57:33.850753 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:57:33.850753 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:57:33.850753 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:57:33.850753 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:57:33.850753 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:57:33.850753 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:57:33.850753 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:57:33.850753 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:57:33.850753 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 13 15:57:33.850753 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 13 15:57:33.850753 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 13 15:57:33.850753 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 13 15:57:33.851900 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:57:33.851900 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:57:33.851900 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:57:33.851900 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:57:33.851900 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:57:33.851900 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:57:33.851900 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:57:33.851900 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application with Lock( Sep 13 15:57:33.851900 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:57:33.851900 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:57:33.851900 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:57:33.851900 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:57:33.851900 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:57:33.851900 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:57:33.851900 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:57:33.851900 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:57:33.851900 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 13 15:57:33.853043 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 13 15:57:33.853043 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:57:33.853043 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application Sep 13 15:57:33.853043 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 651/1283] 10.222.0.50 () {56 vars in 1108 bytes} [Tue Sep 13 15:57:33 2022] GET /identity/v3/domains/Default => generated 87 bytes in 163 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:57:33.859413 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab36b840-69f9-45a0-afb3-f5324729146c None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:57:33.955907 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab36b840-69f9-45a0-afb3-f5324729146c None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:57:33.988304 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f552c45-1a37-4c5a-a967-b865eefe8ce7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:57:33.988896 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f552c45-1a37-4c5a-a967-b865eefe8ce7 None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:57:33.989175 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f552c45-1a37-4c5a-a967-b865eefe8ce7 None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:57:33.989447 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f552c45-1a37-4c5a-a967-b865eefe8ce7 None admin] PATH_INFO: `/v3/domains` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:57:33.990076 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f552c45-1a37-4c5a-a967-b865eefe8ce7 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81438) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:57:33.990375 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f552c45-1a37-4c5a-a967-b865eefe8ce7 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:57:33.991753 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f552c45-1a37-4c5a-a967-b865eefe8ce7 None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:57:34.002788 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 633/1284] 10.222.0.50 () {56 vars in 1109 bytes} [Tue Sep 13 15:57:33 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 144 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:57:34.008293 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9daa52be-304c-4b10-ab74-08674b55a11d None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:57:34.107498 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9daa52be-304c-4b10-ab74-08674b55a11d None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:57:34.120686 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:57:34.141174 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42345b55-bc7c-4315-a7cd-8e335ac0ceb0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:57:34.141792 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42345b55-bc7c-4315-a7cd-8e335ac0ceb0 None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:57:34.142073 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42345b55-bc7c-4315-a7cd-8e335ac0ceb0 None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:57:34.142347 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42345b55-bc7c-4315-a7cd-8e335ac0ceb0 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:57:34.148796 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42345b55-bc7c-4315-a7cd-8e335ac0ceb0 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:57:34.150241 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42345b55-bc7c-4315-a7cd-8e335ac0ceb0 None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:57:34.161881 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application [None req-42345b55-bc7c-4315-a7cd-8e335ac0ceb0 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 13 15:57:34.161881 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:57:34.161881 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:57:34.161881 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:57:34.161881 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:57:34.161881 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:57:34.161881 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:57:34.161881 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:57:34.161881 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:57:34.161881 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:57:34.161881 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:57:34.161881 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:57:34.161881 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 13 15:57:34.161881 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 13 15:57:34.161881 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 13 15:57:34.162981 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 13 15:57:34.162981 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:57:34.162981 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:57:34.162981 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:57:34.162981 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:57:34.162981 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:57:34.162981 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:57:34.162981 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:57:34.162981 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application with Lock( Sep 13 15:57:34.162981 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:57:34.162981 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:57:34.162981 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:57:34.162981 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:57:34.162981 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:57:34.162981 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:57:34.162981 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:57:34.162981 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:57:34.164064 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 13 15:57:34.164064 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 13 15:57:34.164064 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 13 15:57:34.164064 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 13 15:57:34.164064 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 13 15:57:34.164064 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 13 15:57:34.164064 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 13 15:57:34.164064 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: ERROR keystone.server.flask.application Sep 13 15:57:34.164064 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 652/1285] 10.222.0.50 () {56 vars in 1111 bytes} [Tue Sep 13 15:57:34 2022] GET /identity/v3/projects/service => generated 88 bytes in 155 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:57:34.169008 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f552c45-1a37-4c5a-a967-b865eefe8ce7 None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:57:34.218779 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:57:34.267131 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f552c45-1a37-4c5a-a967-b865eefe8ce7 None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:57:34.298205 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6701e11b-f442-4015-8f4a-fa2ef218e5ae None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:57:34.298801 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6701e11b-f442-4015-8f4a-fa2ef218e5ae None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:57:34.299102 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6701e11b-f442-4015-8f4a-fa2ef218e5ae None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:57:34.299368 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6701e11b-f442-4015-8f4a-fa2ef218e5ae None admin] PATH_INFO: `/v3/projects` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:57:34.300037 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6701e11b-f442-4015-8f4a-fa2ef218e5ae None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81438) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:57:34.300334 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6701e11b-f442-4015-8f4a-fa2ef218e5ae None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:57:34.301983 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6701e11b-f442-4015-8f4a-fa2ef218e5ae None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:57:34.314012 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 634/1286] 10.222.0.50 () {56 vars in 1148 bytes} [Tue Sep 13 15:57:34 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 145 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:57:34.319986 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42345b55-bc7c-4315-a7cd-8e335ac0ceb0 None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:57:34.416923 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42345b55-bc7c-4315-a7cd-8e335ac0ceb0 None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:57:34.449646 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82726e3a-c973-4e73-a194-8a7ff665ecf7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:57:34.450242 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82726e3a-c973-4e73-a194-8a7ff665ecf7 None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:57:34.450524 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82726e3a-c973-4e73-a194-8a7ff665ecf7 None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:57:34.450800 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82726e3a-c973-4e73-a194-8a7ff665ecf7 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:57:34.451550 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82726e3a-c973-4e73-a194-8a7ff665ecf7 None admin] RBAC: Adding query filter params (role.id=031d4e1a23e34f71858fc4a6dfbd4ccb, scope.project.id=5811e3b83a1e462e9320cad79a3ad791, user.id=31480b61418340bd88583f034f67412e) {{(pid=81437) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:57:34.451895 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82726e3a-c973-4e73-a194-8a7ff665ecf7 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:57:34.453965 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82726e3a-c973-4e73-a194-8a7ff665ecf7 None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:57:34.459543 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 653/1287] 10.222.0.50 () {56 vars in 1374 bytes} [Tue Sep 13 15:57:34 2022] GET /identity/v3/role_assignments?user.id=31480b61418340bd88583f034f67412e&scope.project.id=5811e3b83a1e462e9320cad79a3ad791&role.id=031d4e1a23e34f71858fc4a6dfbd4ccb => generated 608 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:57:36.586012 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6866389b-2d5e-42de-9772-381d635bc68c None None] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:57:36.586586 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6866389b-2d5e-42de-9772-381d635bc68c None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:57:36.586586 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6866389b-2d5e-42de-9772-381d635bc68c None None] PATH_INFO: `/` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:57:36.587934 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 635/1288] 10.222.0.50 () {54 vars in 927 bytes} [Tue Sep 13 15:57:36 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:57:36.592992 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce513983-2039-45aa-af8f-8931d911861d None None] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:57:36.593372 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce513983-2039-45aa-af8f-8931d911861d None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:57:36.593540 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce513983-2039-45aa-af8f-8931d911861d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:57:36.611266 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.auth.core [None req-ce513983-2039-45aa-af8f-8931d911861d None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81437) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:57:36.612426 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-ce513983-2039-45aa-af8f-8931d911861d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:57:36.641277 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:57:36.676969 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 654/1289] 10.222.0.50 () {58 vars in 1024 bytes} [Tue Sep 13 15:57:36 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 13 15:57:36.943377 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6866389b-2d5e-42de-9772-381d635bc68c None None] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:57:37.042123 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6866389b-2d5e-42de-9772-381d635bc68c None None] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:57:37.053962 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:57:37.078499 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb58b687-e059-4b02-8c80-86b3d3c1708f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:57:37.079153 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb58b687-e059-4b02-8c80-86b3d3c1708f None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:57:37.079431 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb58b687-e059-4b02-8c80-86b3d3c1708f None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:57:37.079809 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb58b687-e059-4b02-8c80-86b3d3c1708f None admin] PATH_INFO: `/v3/services/placement` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:57:37.080381 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb58b687-e059-4b02-8c80-86b3d3c1708f None admin] RBAC: Authorizing `identity:get_service(service_id=placement)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:57:37.081726 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb58b687-e059-4b02-8c80-86b3d3c1708f None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:57:37.090168 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application [None req-eb58b687-e059-4b02-8c80-86b3d3c1708f None admin] Could not find service: placement.: keystone.exception.ServiceNotFound: Could not find service: placement. Sep 13 15:57:37.090168 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:57:37.090168 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 13 15:57:37.090168 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 13 15:57:37.090168 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 13 15:57:37.090168 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 13 15:57:37.090168 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 13 15:57:37.090168 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 13 15:57:37.090168 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Sep 13 15:57:37.090168 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Sep 13 15:57:37.090168 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 13 15:57:37.090168 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Sep 13 15:57:37.090168 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:57:37.090168 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:57:37.090168 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:57:37.090168 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:57:37.090168 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:57:37.091389 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:57:37.091389 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:57:37.091389 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:57:37.091389 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:57:37.091389 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:57:37.091389 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:57:37.091389 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 13 15:57:37.091389 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 13 15:57:37.091389 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 13 15:57:37.091389 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 13 15:57:37.091389 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:57:37.091389 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:57:37.091389 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:57:37.091389 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:57:37.091389 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:57:37.091389 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:57:37.092271 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:57:37.092271 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application with Lock( Sep 13 15:57:37.092271 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:57:37.092271 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:57:37.092271 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:57:37.092271 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:57:37.092271 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:57:37.092271 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:57:37.092271 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:57:37.092271 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:57:37.092271 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 13 15:57:37.092271 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 13 15:57:37.092271 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Sep 13 15:57:37.092271 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Sep 13 15:57:37.092271 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 636/1290] 10.222.0.50 () {56 vars in 1117 bytes} [Tue Sep 13 15:57:36 2022] GET /identity/v3/services/placement => generated 90 bytes in 149 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:57:37.097623 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce513983-2039-45aa-af8f-8931d911861d None None] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:57:37.199243 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce513983-2039-45aa-af8f-8931d911861d None None] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:57:37.231749 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7430b768-3b11-4f6d-89fa-918cb18ba369 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:57:37.232339 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7430b768-3b11-4f6d-89fa-918cb18ba369 None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:57:37.232615 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7430b768-3b11-4f6d-89fa-918cb18ba369 None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:57:37.233582 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7430b768-3b11-4f6d-89fa-918cb18ba369 None admin] PATH_INFO: `/v3/services` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:57:37.234211 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7430b768-3b11-4f6d-89fa-918cb18ba369 None admin] RBAC: Adding query filter params (name=placement) {{(pid=81437) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:57:37.234513 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7430b768-3b11-4f6d-89fa-918cb18ba369 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:57:37.235894 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7430b768-3b11-4f6d-89fa-918cb18ba369 None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:57:37.240518 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 655/1291] 10.222.0.50 () {56 vars in 1116 bytes} [Tue Sep 13 15:57:37 2022] GET /identity/v3/services?name=placement => generated 126 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:57:37.246379 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb58b687-e059-4b02-8c80-86b3d3c1708f None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:57:37.344230 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb58b687-e059-4b02-8c80-86b3d3c1708f None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:57:37.376857 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4806ac2-5f84-4e50-8674-30ff8de16474 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:57:37.377434 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4806ac2-5f84-4e50-8674-30ff8de16474 None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:57:37.377708 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4806ac2-5f84-4e50-8674-30ff8de16474 None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:57:37.377976 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4806ac2-5f84-4e50-8674-30ff8de16474 None admin] PATH_INFO: `/v3/services` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:57:37.378585 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4806ac2-5f84-4e50-8674-30ff8de16474 None admin] RBAC: Adding query filter params (type=placement) {{(pid=81438) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:57:37.378883 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4806ac2-5f84-4e50-8674-30ff8de16474 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:57:37.380248 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4806ac2-5f84-4e50-8674-30ff8de16474 None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:57:37.384490 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 637/1292] 10.222.0.50 () {56 vars in 1116 bytes} [Tue Sep 13 15:57:37 2022] GET /identity/v3/services?type=placement => generated 126 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:57:39.482788 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1edc1364-f602-4629-824b-d3cf04e9cfcd None None] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:57:39.483354 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1edc1364-f602-4629-824b-d3cf04e9cfcd None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:57:39.483354 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1edc1364-f602-4629-824b-d3cf04e9cfcd None None] PATH_INFO: `/` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:57:39.485312 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 656/1293] 10.222.0.50 () {54 vars in 927 bytes} [Tue Sep 13 15:57:39 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:57:39.490572 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63ec0f83-731f-412d-8714-2c5a9d02cf19 None None] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:57:39.491001 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63ec0f83-731f-412d-8714-2c5a9d02cf19 None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:57:39.491156 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63ec0f83-731f-412d-8714-2c5a9d02cf19 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:57:39.509296 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.auth.core [None req-63ec0f83-731f-412d-8714-2c5a9d02cf19 None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81438) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:57:39.510458 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-63ec0f83-731f-412d-8714-2c5a9d02cf19 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:57:39.539336 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:57:39.576432 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 638/1294] 10.222.0.50 () {58 vars in 1024 bytes} [Tue Sep 13 15:57:39 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 13 15:57:39.842036 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1edc1364-f602-4629-824b-d3cf04e9cfcd None None] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:57:39.872832 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:57:39.942390 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1edc1364-f602-4629-824b-d3cf04e9cfcd None None] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:57:39.976353 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca5a5408-274d-4550-a046-391939f21227 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:57:39.976946 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca5a5408-274d-4550-a046-391939f21227 None admin] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:57:39.977226 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca5a5408-274d-4550-a046-391939f21227 None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:57:39.977499 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca5a5408-274d-4550-a046-391939f21227 None admin] PATH_INFO: `/v3/services` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:57:39.978140 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca5a5408-274d-4550-a046-391939f21227 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:57:39.979552 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca5a5408-274d-4550-a046-391939f21227 None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:57:39.993252 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 657/1295] 10.222.0.50 () {60 vars in 1141 bytes} [Tue Sep 13 15:57:39 2022] POST /identity/v3/services => generated 246 bytes in 152 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 15:57:42.152237 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fde3e553-1f0f-49fa-afb3-156b4c12c539 None None] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:57:42.152744 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fde3e553-1f0f-49fa-afb3-156b4c12c539 None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:57:42.152833 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fde3e553-1f0f-49fa-afb3-156b4c12c539 None None] PATH_INFO: `/` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:57:42.154109 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 639/1296] 10.222.0.50 () {54 vars in 927 bytes} [Tue Sep 13 15:57:42 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:57:42.159674 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d79ac0e6-6f51-411d-9bd6-b5701f9aa622 None None] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:57:42.160077 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d79ac0e6-6f51-411d-9bd6-b5701f9aa622 None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:57:42.160217 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d79ac0e6-6f51-411d-9bd6-b5701f9aa622 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:57:42.178530 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.auth.core [None req-d79ac0e6-6f51-411d-9bd6-b5701f9aa622 None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81437) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:57:42.179723 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-d79ac0e6-6f51-411d-9bd6-b5701f9aa622 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:57:42.249037 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 658/1297] 10.222.0.50 () {58 vars in 1024 bytes} [Tue Sep 13 15:57:42 2022] POST /identity/v3/auth/tokens => generated 2221 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 13 15:57:42.514110 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fde3e553-1f0f-49fa-afb3-156b4c12c539 None None] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:57:42.611777 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fde3e553-1f0f-49fa-afb3-156b4c12c539 None None] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:57:42.644319 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2128fd9-d0de-4930-bb0d-fba5f4176eb2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:57:42.644926 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2128fd9-d0de-4930-bb0d-fba5f4176eb2 None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:57:42.645204 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2128fd9-d0de-4930-bb0d-fba5f4176eb2 None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:57:42.645470 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2128fd9-d0de-4930-bb0d-fba5f4176eb2 None admin] PATH_INFO: `/v3/services/placement` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:57:42.646033 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2128fd9-d0de-4930-bb0d-fba5f4176eb2 None admin] RBAC: Authorizing `identity:get_service(service_id=placement)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:57:42.647412 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2128fd9-d0de-4930-bb0d-fba5f4176eb2 None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:57:42.656294 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application [None req-c2128fd9-d0de-4930-bb0d-fba5f4176eb2 None admin] Could not find service: placement.: keystone.exception.ServiceNotFound: Could not find service: placement. Sep 13 15:57:42.656294 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:57:42.656294 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 13 15:57:42.656294 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 13 15:57:42.656294 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 13 15:57:42.656294 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 13 15:57:42.656294 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 13 15:57:42.656294 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 13 15:57:42.656294 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Sep 13 15:57:42.656294 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Sep 13 15:57:42.656294 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 13 15:57:42.656294 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Sep 13 15:57:42.656294 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:57:42.656294 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:57:42.656294 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:57:42.656294 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:57:42.656294 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:57:42.657578 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:57:42.657578 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:57:42.657578 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:57:42.657578 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:57:42.657578 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:57:42.657578 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:57:42.657578 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 13 15:57:42.657578 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 13 15:57:42.657578 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 13 15:57:42.657578 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 13 15:57:42.657578 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:57:42.657578 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:57:42.657578 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:57:42.657578 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:57:42.657578 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:57:42.657578 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:57:42.658454 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:57:42.658454 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application with Lock( Sep 13 15:57:42.658454 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:57:42.658454 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:57:42.658454 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:57:42.658454 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:57:42.658454 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:57:42.658454 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:57:42.658454 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:57:42.658454 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:57:42.658454 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 13 15:57:42.658454 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 13 15:57:42.658454 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Sep 13 15:57:42.658454 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Sep 13 15:57:42.658454 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 640/1298] 10.222.0.50 () {56 vars in 1117 bytes} [Tue Sep 13 15:57:42 2022] GET /identity/v3/services/placement => generated 90 bytes in 144 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:57:42.663229 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d79ac0e6-6f51-411d-9bd6-b5701f9aa622 None None] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:57:42.681992 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:57:42.765529 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d79ac0e6-6f51-411d-9bd6-b5701f9aa622 None None] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:57:42.801163 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40b399f3-875c-43ba-a6c7-870a5debe16f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:57:42.801769 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40b399f3-875c-43ba-a6c7-870a5debe16f None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:57:42.802046 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40b399f3-875c-43ba-a6c7-870a5debe16f None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:57:42.802316 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40b399f3-875c-43ba-a6c7-870a5debe16f None admin] PATH_INFO: `/v3/services` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:57:42.802961 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40b399f3-875c-43ba-a6c7-870a5debe16f None admin] RBAC: Adding query filter params (name=placement) {{(pid=81437) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:57:42.803266 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40b399f3-875c-43ba-a6c7-870a5debe16f None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:57:42.804629 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40b399f3-875c-43ba-a6c7-870a5debe16f None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:57:42.809667 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 659/1299] 10.222.0.50 () {56 vars in 1116 bytes} [Tue Sep 13 15:57:42 2022] GET /identity/v3/services?name=placement => generated 358 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:57:42.815919 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2128fd9-d0de-4930-bb0d-fba5f4176eb2 None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:57:42.847572 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:57:42.914431 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2128fd9-d0de-4930-bb0d-fba5f4176eb2 None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:57:42.946549 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46f0e664-7b0d-4d02-8c1d-5d0dfa63902e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:57:42.947166 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46f0e664-7b0d-4d02-8c1d-5d0dfa63902e None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:57:42.947445 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46f0e664-7b0d-4d02-8c1d-5d0dfa63902e None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:57:42.947713 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46f0e664-7b0d-4d02-8c1d-5d0dfa63902e None admin] PATH_INFO: `/v3/endpoints` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:57:42.948395 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46f0e664-7b0d-4d02-8c1d-5d0dfa63902e None admin] RBAC: Adding query filter params (interface=public, service_id=1027c04db48a4f9e807e2c41e08a1380, region_id=RegionOne) {{(pid=81438) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:57:42.948699 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46f0e664-7b0d-4d02-8c1d-5d0dfa63902e None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:57:42.950045 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46f0e664-7b0d-4d02-8c1d-5d0dfa63902e None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:57:42.955368 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 641/1300] 10.222.0.50 () {56 vars in 1251 bytes} [Tue Sep 13 15:57:42 2022] GET /identity/v3/endpoints?service_id=1027c04db48a4f9e807e2c41e08a1380&interface=public®ion_id=RegionOne => generated 194 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:57:42.961071 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40b399f3-875c-43ba-a6c7-870a5debe16f None admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:57:43.040052 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:57:43.058083 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40b399f3-875c-43ba-a6c7-870a5debe16f None admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:57:43.091246 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5761c45f-841b-49e9-8dd5-43848d90e146 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:57:43.091846 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5761c45f-841b-49e9-8dd5-43848d90e146 None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:57:43.092122 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5761c45f-841b-49e9-8dd5-43848d90e146 None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:57:43.092393 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5761c45f-841b-49e9-8dd5-43848d90e146 None admin] PATH_INFO: `/v3/services` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:57:43.093531 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5761c45f-841b-49e9-8dd5-43848d90e146 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:57:43.094917 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5761c45f-841b-49e9-8dd5-43848d90e146 None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:57:43.101487 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 660/1301] 10.222.0.50 () {56 vars in 1087 bytes} [Tue Sep 13 15:57:42 2022] GET /identity/v3/services => generated 2191 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 15:57:45.224889 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e339278d-cbc9-44aa-841e-ee84cbdff8fb None None] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:57:45.225459 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e339278d-cbc9-44aa-841e-ee84cbdff8fb None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:57:45.225459 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e339278d-cbc9-44aa-841e-ee84cbdff8fb None None] PATH_INFO: `/` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:57:45.226763 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 642/1302] 10.222.0.50 () {54 vars in 927 bytes} [Tue Sep 13 15:57:45 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:57:45.231937 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a19218e6-3eca-4209-a308-bd6448422e9c None None] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:57:45.232357 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a19218e6-3eca-4209-a308-bd6448422e9c None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:57:45.232467 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a19218e6-3eca-4209-a308-bd6448422e9c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:57:45.250497 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.auth.core [None req-a19218e6-3eca-4209-a308-bd6448422e9c None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81437) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:57:45.251699 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-a19218e6-3eca-4209-a308-bd6448422e9c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:57:45.316279 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 661/1303] 10.222.0.50 () {58 vars in 1024 bytes} [Tue Sep 13 15:57:45 2022] POST /identity/v3/auth/tokens => generated 2221 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 13 15:57:45.593101 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e339278d-cbc9-44aa-841e-ee84cbdff8fb None None] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:57:45.685452 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:57:45.693086 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e339278d-cbc9-44aa-841e-ee84cbdff8fb None None] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:57:45.727084 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05c82fca-e72b-48e0-9520-e817cda61970 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:57:45.727749 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05c82fca-e72b-48e0-9520-e817cda61970 None admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:57:45.728074 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05c82fca-e72b-48e0-9520-e817cda61970 None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:57:45.728407 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05c82fca-e72b-48e0-9520-e817cda61970 None admin] PATH_INFO: `/v3/services/placement` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:57:45.728988 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05c82fca-e72b-48e0-9520-e817cda61970 None admin] RBAC: Authorizing `identity:get_service(service_id=placement)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:57:45.730676 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05c82fca-e72b-48e0-9520-e817cda61970 None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:57:45.739951 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application [None req-05c82fca-e72b-48e0-9520-e817cda61970 None admin] Could not find service: placement.: keystone.exception.ServiceNotFound: Could not find service: placement. Sep 13 15:57:45.739951 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:57:45.739951 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 13 15:57:45.739951 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 13 15:57:45.739951 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 13 15:57:45.739951 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 13 15:57:45.739951 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 13 15:57:45.739951 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 13 15:57:45.739951 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Sep 13 15:57:45.739951 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Sep 13 15:57:45.739951 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 13 15:57:45.739951 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Sep 13 15:57:45.739951 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:57:45.739951 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:57:45.739951 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:57:45.739951 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:57:45.739951 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:57:45.741315 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:57:45.741315 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:57:45.741315 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:57:45.741315 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:57:45.741315 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:57:45.741315 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:57:45.741315 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 13 15:57:45.741315 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 13 15:57:45.741315 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 13 15:57:45.741315 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 13 15:57:45.741315 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:57:45.741315 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:57:45.741315 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:57:45.741315 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:57:45.741315 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:57:45.741315 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:57:45.742189 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:57:45.742189 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application with Lock( Sep 13 15:57:45.742189 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:57:45.742189 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:57:45.742189 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:57:45.742189 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:57:45.742189 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:57:45.742189 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:57:45.742189 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:57:45.742189 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:57:45.742189 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 13 15:57:45.742189 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 13 15:57:45.742189 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Sep 13 15:57:45.742189 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: ERROR keystone.server.flask.application Sep 13 15:57:45.742189 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 643/1304] 10.222.0.50 () {56 vars in 1117 bytes} [Tue Sep 13 15:57:45 2022] GET /identity/v3/services/placement => generated 90 bytes in 149 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:57:45.747328 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a19218e6-3eca-4209-a308-bd6448422e9c None None] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:57:45.822562 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:57:45.845327 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a19218e6-3eca-4209-a308-bd6448422e9c None None] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:57:45.878236 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dad4c693-d819-4da4-bedd-e54586b87b12 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:57:45.878834 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dad4c693-d819-4da4-bedd-e54586b87b12 None admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:57:45.879137 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dad4c693-d819-4da4-bedd-e54586b87b12 None admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:57:45.879407 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dad4c693-d819-4da4-bedd-e54586b87b12 None admin] PATH_INFO: `/v3/services` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:57:45.880020 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dad4c693-d819-4da4-bedd-e54586b87b12 None admin] RBAC: Adding query filter params (name=placement) {{(pid=81437) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:57:45.880320 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dad4c693-d819-4da4-bedd-e54586b87b12 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:57:45.881671 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dad4c693-d819-4da4-bedd-e54586b87b12 None admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:57:45.886605 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 662/1305] 10.222.0.50 () {56 vars in 1116 bytes} [Tue Sep 13 15:57:45 2022] GET /identity/v3/services?name=placement => generated 358 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 13 15:57:45.892513 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05c82fca-e72b-48e0-9520-e817cda61970 None admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:57:45.988601 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05c82fca-e72b-48e0-9520-e817cda61970 None admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:57:46.021058 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-205fccc2-adda-4fcd-b755-4693e6acf860 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:57:46.021641 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-205fccc2-adda-4fcd-b755-4693e6acf860 None admin] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:57:46.021920 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-205fccc2-adda-4fcd-b755-4693e6acf860 None admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:57:46.022193 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-205fccc2-adda-4fcd-b755-4693e6acf860 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:57:46.022840 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-205fccc2-adda-4fcd-b755-4693e6acf860 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:57:46.024213 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-205fccc2-adda-4fcd-b755-4693e6acf860 None admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:57:46.042526 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 644/1306] 10.222.0.50 () {60 vars in 1144 bytes} [Tue Sep 13 15:57:45 2022] POST /identity/v3/endpoints => generated 331 bytes in 150 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 15:57:55.883725 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=30 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:57:56.038301 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=91,keystone:INSERT=1 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:58:49.554546 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6f0a44c-3711-47ca-856e-48b9457d25af None None] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:58:49.555167 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6f0a44c-3711-47ca-856e-48b9457d25af None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:58:49.555167 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6f0a44c-3711-47ca-856e-48b9457d25af None None] PATH_INFO: `/` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:58:49.556465 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 663/1307] 10.222.0.50 () {54 vars in 927 bytes} [Tue Sep 13 15:58:49 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:58:49.561877 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca619de9-1083-4232-8549-74fa61915475 None None] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:58:49.562339 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca619de9-1083-4232-8549-74fa61915475 None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:58:49.562433 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca619de9-1083-4232-8549-74fa61915475 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:58:49.658330 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.auth.core [None req-ca619de9-1083-4232-8549-74fa61915475 None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81438) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:58:49.660287 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-ca619de9-1083-4232-8549-74fa61915475 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:58:49.849262 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 645/1308] 10.222.0.50 () {58 vars in 1024 bytes} [Tue Sep 13 15:58:49 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 288 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:58:50.148514 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71051e8c-b988-461a-8b16-f90d2d49c1b9 None None] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:58:50.148855 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71051e8c-b988-461a-8b16-f90d2d49c1b9 None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:58:50.149072 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71051e8c-b988-461a-8b16-f90d2d49c1b9 None None] PATH_INFO: `/` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:58:50.150439 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 664/1309] 10.222.0.50 () {54 vars in 965 bytes} [Tue Sep 13 15:58:50 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:58:50.157209 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75de602a-0af3-437e-bf55-3729bf87fd38 None None] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:58:50.157672 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75de602a-0af3-437e-bf55-3729bf87fd38 None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:58:50.158042 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75de602a-0af3-437e-bf55-3729bf87fd38 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:58:50.255720 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.auth.core [None req-75de602a-0af3-437e-bf55-3729bf87fd38 None None] MFA Rules not processed for user `2f436f07939d4855a21b195bc7566244`. Rule list: `[]` (Enabled: `True`). {{(pid=81438) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:58:50.257033 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-75de602a-0af3-437e-bf55-3729bf87fd38 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:58:50.280298 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:58:50.345064 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 646/1310] 10.222.0.50 () {58 vars in 1062 bytes} [Tue Sep 13 15:58:50 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 189 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:58:50.354717 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71051e8c-b988-461a-8b16-f90d2d49c1b9 None None] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:58:50.356772 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-71051e8c-b988-461a-8b16-f90d2d49c1b9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:58:50.474235 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71051e8c-b988-461a-8b16-f90d2d49c1b9 None None] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:58:50.552607 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3240c654-6f71-4d3d-98d3-f1f21c44c157 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f436f07939d4855a21b195bc7566244', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:58:50.553225 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3240c654-6f71-4d3d-98d3-f1f21c44c157 service neutron] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:58:50.553506 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3240c654-6f71-4d3d-98d3-f1f21c44c157 service neutron] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:58:50.553783 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3240c654-6f71-4d3d-98d3-f1f21c44c157 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:58:50.555422 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-3240c654-6f71-4d3d-98d3-f1f21c44c157 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:58:50.623009 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:58:50.687868 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3240c654-6f71-4d3d-98d3-f1f21c44c157 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:58:50.689306 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3240c654-6f71-4d3d-98d3-f1f21c44c157 service neutron] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:58:50.709490 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 665/1311] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 15:58:50 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 356 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 15:58:53.178055 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0782c9ee-5b68-4eb3-804d-213bab0433b2 None None] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:58:53.178604 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0782c9ee-5b68-4eb3-804d-213bab0433b2 None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:58:53.178604 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0782c9ee-5b68-4eb3-804d-213bab0433b2 None None] PATH_INFO: `/` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:58:53.180044 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 647/1312] 10.222.0.50 () {54 vars in 927 bytes} [Tue Sep 13 15:58:53 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:58:53.186184 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d83aa55-5561-4126-ae33-d8f65604ea17 None None] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:58:53.186693 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d83aa55-5561-4126-ae33-d8f65604ea17 None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:58:53.186791 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d83aa55-5561-4126-ae33-d8f65604ea17 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:58:53.259358 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.auth.core [None req-1d83aa55-5561-4126-ae33-d8f65604ea17 None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81437) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:58:53.261173 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-1d83aa55-5561-4126-ae33-d8f65604ea17 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:58:53.357521 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 666/1313] 10.222.0.50 () {58 vars in 1024 bytes} [Tue Sep 13 15:58:53 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 172 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:58:53.640212 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0782c9ee-5b68-4eb3-804d-213bab0433b2 None None] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:58:53.641739 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-0782c9ee-5b68-4eb3-804d-213bab0433b2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:58:53.746182 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0782c9ee-5b68-4eb3-804d-213bab0433b2 None None] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:58:53.818873 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:58:53.821167 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68ea826e-f3c4-44f1-b1c0-ebf09c13acc1 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f436f07939d4855a21b195bc7566244', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:58:53.821812 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68ea826e-f3c4-44f1-b1c0-ebf09c13acc1 service neutron] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:58:53.822485 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68ea826e-f3c4-44f1-b1c0-ebf09c13acc1 service neutron] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:58:53.823064 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68ea826e-f3c4-44f1-b1c0-ebf09c13acc1 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:58:53.825174 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-68ea826e-f3c4-44f1-b1c0-ebf09c13acc1 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:58:53.939738 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68ea826e-f3c4-44f1-b1c0-ebf09c13acc1 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:58:53.941219 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68ea826e-f3c4-44f1-b1c0-ebf09c13acc1 service neutron] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:58:53.958922 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 648/1314] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 15:58:53 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 319 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 13 15:58:56.408909 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36b86abd-522f-4ecf-b757-b9dff3398911 None None] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:58:56.409442 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36b86abd-522f-4ecf-b757-b9dff3398911 None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:58:56.409442 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36b86abd-522f-4ecf-b757-b9dff3398911 None None] PATH_INFO: `/` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:58:56.410791 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 667/1315] 10.222.0.50 () {54 vars in 927 bytes} [Tue Sep 13 15:58:56 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:58:56.416446 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c50f0a27-a6ef-4bce-b8f8-59e3999ac4ce None None] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:58:56.416859 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c50f0a27-a6ef-4bce-b8f8-59e3999ac4ce None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:58:56.417002 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c50f0a27-a6ef-4bce-b8f8-59e3999ac4ce None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:58:56.482242 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.auth.core [None req-c50f0a27-a6ef-4bce-b8f8-59e3999ac4ce None None] MFA Rules not processed for user `5c4340db782249918182a2e21023cc55`. Rule list: `[]` (Enabled: `True`). {{(pid=81438) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:58:56.483467 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-c50f0a27-a6ef-4bce-b8f8-59e3999ac4ce None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:58:56.534511 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:58:56.580088 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 649/1316] 10.222.0.50 () {58 vars in 1024 bytes} [Tue Sep 13 15:58:56 2022] POST /identity/v3/auth/tokens => generated 3243 bytes in 165 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:58:56.889270 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f97c7c0e-202c-493c-910f-339e054c9478 None None] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:58:56.889551 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f97c7c0e-202c-493c-910f-339e054c9478 None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:58:56.889823 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f97c7c0e-202c-493c-910f-339e054c9478 None None] PATH_INFO: `/` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:58:56.891220 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 668/1317] 10.222.0.50 () {54 vars in 965 bytes} [Tue Sep 13 15:58:56 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:58:56.896941 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8643ec8-b2b4-42b6-8770-3702cd8fa6ca None None] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:58:56.897224 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8643ec8-b2b4-42b6-8770-3702cd8fa6ca None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:58:56.897494 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8643ec8-b2b4-42b6-8770-3702cd8fa6ca None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:58:56.974853 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.auth.core [None req-d8643ec8-b2b4-42b6-8770-3702cd8fa6ca None None] MFA Rules not processed for user `2f436f07939d4855a21b195bc7566244`. Rule list: `[]` (Enabled: `True`). {{(pid=81438) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:58:56.976063 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-d8643ec8-b2b4-42b6-8770-3702cd8fa6ca None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:58:57.053806 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 650/1318] 10.222.0.50 () {58 vars in 1062 bytes} [Tue Sep 13 15:58:56 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 158 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:58:57.063312 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f97c7c0e-202c-493c-910f-339e054c9478 None None] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:58:57.064831 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-f97c7c0e-202c-493c-910f-339e054c9478 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:58:57.077239 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:58:57.169575 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f97c7c0e-202c-493c-910f-339e054c9478 None None] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:58:57.240971 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f0f5134-aa45-4864-848c-68603b5b60b8 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f436f07939d4855a21b195bc7566244', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:58:57.241572 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f0f5134-aa45-4864-848c-68603b5b60b8 service neutron] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:58:57.241852 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f0f5134-aa45-4864-848c-68603b5b60b8 service neutron] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:58:57.242125 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f0f5134-aa45-4864-848c-68603b5b60b8 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:58:57.243741 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-2f0f5134-aa45-4864-848c-68603b5b60b8 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:58:57.305000 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:58:57.351697 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f0f5134-aa45-4864-848c-68603b5b60b8 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:58:57.353130 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f0f5134-aa45-4864-848c-68603b5b60b8 service neutron] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:58:57.371646 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 669/1319] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 15:58:57 2022] GET /identity/v3/auth/tokens => generated 3243 bytes in 309 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 15:59:00.393709 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-012d3a4f-3473-4073-afe2-b09c0e609041 None None] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:59:00.394282 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-012d3a4f-3473-4073-afe2-b09c0e609041 None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:59:00.394282 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-012d3a4f-3473-4073-afe2-b09c0e609041 None None] PATH_INFO: `/` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:59:00.395660 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 651/1320] 10.222.0.50 () {54 vars in 927 bytes} [Tue Sep 13 15:59:00 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:59:00.401317 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7197ccec-0211-4857-998b-9b9405cf15ba None None] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:59:00.401704 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7197ccec-0211-4857-998b-9b9405cf15ba None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:59:00.401865 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7197ccec-0211-4857-998b-9b9405cf15ba None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:59:00.467081 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.auth.core [None req-7197ccec-0211-4857-998b-9b9405cf15ba None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81437) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:59:00.468275 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-7197ccec-0211-4857-998b-9b9405cf15ba None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:59:00.563214 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 670/1321] 10.222.0.50 () {58 vars in 1024 bytes} [Tue Sep 13 15:59:00 2022] POST /identity/v3/auth/tokens => generated 3238 bytes in 163 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:59:00.847380 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-012d3a4f-3473-4073-afe2-b09c0e609041 None None] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:59:00.848920 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-012d3a4f-3473-4073-afe2-b09c0e609041 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:59:00.901177 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:59:00.954797 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-012d3a4f-3473-4073-afe2-b09c0e609041 None None] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:59:01.027346 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8bad06d-cd14-4027-9aad-c71b6c4d577a service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f436f07939d4855a21b195bc7566244', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:59:01.027945 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8bad06d-cd14-4027-9aad-c71b6c4d577a service neutron] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:59:01.028247 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8bad06d-cd14-4027-9aad-c71b6c4d577a service neutron] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:59:01.028518 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8bad06d-cd14-4027-9aad-c71b6c4d577a service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:59:01.030088 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-a8bad06d-cd14-4027-9aad-c71b6c4d577a service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:59:01.127680 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:59:01.141193 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8bad06d-cd14-4027-9aad-c71b6c4d577a service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:59:01.142630 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8bad06d-cd14-4027-9aad-c71b6c4d577a service neutron] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:59:01.160738 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 652/1322] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 15:59:00 2022] GET /identity/v3/auth/tokens => generated 3238 bytes in 314 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 15:59:04.015090 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08f31218-314f-4e10-919c-60c45bebe6c4 None None] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:59:04.015651 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08f31218-314f-4e10-919c-60c45bebe6c4 None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:59:04.015651 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08f31218-314f-4e10-919c-60c45bebe6c4 None None] PATH_INFO: `/` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:59:04.017000 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 671/1323] 10.222.0.50 () {54 vars in 927 bytes} [Tue Sep 13 15:59:04 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:59:04.022322 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c45ca451-354a-4114-bfab-9fdbf002f4ec None None] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:59:04.022751 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c45ca451-354a-4114-bfab-9fdbf002f4ec None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:59:04.022867 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c45ca451-354a-4114-bfab-9fdbf002f4ec None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:59:04.100321 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.auth.core [None req-c45ca451-354a-4114-bfab-9fdbf002f4ec None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81438) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:59:04.101554 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-c45ca451-354a-4114-bfab-9fdbf002f4ec None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:59:04.213285 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 653/1324] 10.222.0.50 () {58 vars in 1024 bytes} [Tue Sep 13 15:59:04 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 192 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:59:04.502675 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08f31218-314f-4e10-919c-60c45bebe6c4 None None] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:59:04.504291 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-08f31218-314f-4e10-919c-60c45bebe6c4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:59:04.525816 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:59:04.609934 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08f31218-314f-4e10-919c-60c45bebe6c4 None None] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:59:04.681873 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27da93fc-3c64-4933-bf09-de300a21bc82 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f436f07939d4855a21b195bc7566244', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:59:04.682481 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27da93fc-3c64-4933-bf09-de300a21bc82 service neutron] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:59:04.682756 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27da93fc-3c64-4933-bf09-de300a21bc82 service neutron] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:59:04.683073 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27da93fc-3c64-4933-bf09-de300a21bc82 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:59:04.684635 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-27da93fc-3c64-4933-bf09-de300a21bc82 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:59:04.758125 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:59:04.796278 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27da93fc-3c64-4933-bf09-de300a21bc82 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:59:04.797712 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27da93fc-3c64-4933-bf09-de300a21bc82 service neutron] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:59:04.815999 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 672/1325] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 15:59:04 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 314 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 15:59:07.056659 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c66dd2b6-2d65-47b1-b130-a9d86744aef7 None None] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:59:07.057190 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c66dd2b6-2d65-47b1-b130-a9d86744aef7 None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:59:07.057190 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c66dd2b6-2d65-47b1-b130-a9d86744aef7 None None] PATH_INFO: `/` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:59:07.060814 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 654/1326] 10.222.0.50 () {54 vars in 927 bytes} [Tue Sep 13 15:59:07 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:59:07.063943 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f31039d-5693-438d-8f95-2d7b9d561d9e None None] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:59:07.064323 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f31039d-5693-438d-8f95-2d7b9d561d9e None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:59:07.064481 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f31039d-5693-438d-8f95-2d7b9d561d9e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:59:07.138707 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.auth.core [None req-3f31039d-5693-438d-8f95-2d7b9d561d9e None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81437) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:59:07.140067 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-3f31039d-5693-438d-8f95-2d7b9d561d9e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:59:07.237978 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 673/1327] 10.222.0.50 () {58 vars in 1024 bytes} [Tue Sep 13 15:59:07 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 175 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:59:07.519952 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c66dd2b6-2d65-47b1-b130-a9d86744aef7 None None] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:59:07.521526 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-c66dd2b6-2d65-47b1-b130-a9d86744aef7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:59:07.582325 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:59:07.627051 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c66dd2b6-2d65-47b1-b130-a9d86744aef7 None None] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:59:07.702692 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d3cb216-6b14-4447-bb2f-f820311d8e6e service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f436f07939d4855a21b195bc7566244', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:59:07.703325 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d3cb216-6b14-4447-bb2f-f820311d8e6e service neutron] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:59:07.703598 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d3cb216-6b14-4447-bb2f-f820311d8e6e service neutron] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:59:07.703868 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d3cb216-6b14-4447-bb2f-f820311d8e6e service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:59:07.705424 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-7d3cb216-6b14-4447-bb2f-f820311d8e6e service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:59:07.812784 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:59:07.815540 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d3cb216-6b14-4447-bb2f-f820311d8e6e service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:59:07.817624 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d3cb216-6b14-4447-bb2f-f820311d8e6e service neutron] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:59:07.836591 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 655/1328] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 15:59:07 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 317 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 15:59:10.047912 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd68fad5-67e8-41f9-bccc-556193d5183c None None] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:59:10.048429 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd68fad5-67e8-41f9-bccc-556193d5183c None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:59:10.048429 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd68fad5-67e8-41f9-bccc-556193d5183c None None] PATH_INFO: `/` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:59:10.049766 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 674/1329] 10.222.0.50 () {54 vars in 927 bytes} [Tue Sep 13 15:59:10 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:59:10.054887 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39aea37a-91a0-448d-914d-2ccabd00c55f None None] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:59:10.056239 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39aea37a-91a0-448d-914d-2ccabd00c55f None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:59:10.056239 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39aea37a-91a0-448d-914d-2ccabd00c55f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:59:10.129979 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.auth.core [None req-39aea37a-91a0-448d-914d-2ccabd00c55f None None] MFA Rules not processed for user `5c4340db782249918182a2e21023cc55`. Rule list: `[]` (Enabled: `True`). {{(pid=81438) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:59:10.131154 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-39aea37a-91a0-448d-914d-2ccabd00c55f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:59:10.225548 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 656/1330] 10.222.0.50 () {58 vars in 1024 bytes} [Tue Sep 13 15:59:10 2022] POST /identity/v3/auth/tokens => generated 3243 bytes in 172 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:59:10.511044 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd68fad5-67e8-41f9-bccc-556193d5183c None None] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:59:10.512589 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-dd68fad5-67e8-41f9-bccc-556193d5183c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:59:10.545537 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:59:10.621953 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd68fad5-67e8-41f9-bccc-556193d5183c None None] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:59:10.695376 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2dafce6d-a251-41c3-b0b2-04cdab61881c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f436f07939d4855a21b195bc7566244', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:59:10.696111 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dafce6d-a251-41c3-b0b2-04cdab61881c service neutron] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:59:10.696580 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dafce6d-a251-41c3-b0b2-04cdab61881c service neutron] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:59:10.696856 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dafce6d-a251-41c3-b0b2-04cdab61881c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:59:10.698431 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-2dafce6d-a251-41c3-b0b2-04cdab61881c service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:59:10.775087 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:59:10.808454 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2dafce6d-a251-41c3-b0b2-04cdab61881c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:59:10.809892 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2dafce6d-a251-41c3-b0b2-04cdab61881c service neutron] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:59:10.827825 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 675/1331] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 15:59:10 2022] GET /identity/v3/auth/tokens => generated 3243 bytes in 317 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 15:59:13.239543 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71307c74-caef-462e-9818-7ddb572bcf0f None None] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:59:13.240054 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71307c74-caef-462e-9818-7ddb572bcf0f None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:59:13.240054 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71307c74-caef-462e-9818-7ddb572bcf0f None None] PATH_INFO: `/` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:59:13.241340 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 657/1332] 10.222.0.50 () {54 vars in 927 bytes} [Tue Sep 13 15:59:13 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:59:13.250941 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70ec246a-4719-4ec7-b863-b2f4eb2973bd None None] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:59:13.251363 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70ec246a-4719-4ec7-b863-b2f4eb2973bd None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:59:13.251493 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70ec246a-4719-4ec7-b863-b2f4eb2973bd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:59:13.320373 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.auth.core [None req-70ec246a-4719-4ec7-b863-b2f4eb2973bd None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81437) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:59:13.322013 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-70ec246a-4719-4ec7-b863-b2f4eb2973bd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:59:13.417435 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 676/1333] 10.222.0.50 () {58 vars in 1024 bytes} [Tue Sep 13 15:59:13 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 172 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:59:13.701738 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71307c74-caef-462e-9818-7ddb572bcf0f None None] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:59:13.703347 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-71307c74-caef-462e-9818-7ddb572bcf0f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:59:13.772574 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:59:13.808779 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71307c74-caef-462e-9818-7ddb572bcf0f None None] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:59:13.881798 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01b89500-a22f-4c85-9918-475b0a393304 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f436f07939d4855a21b195bc7566244', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:59:13.882397 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01b89500-a22f-4c85-9918-475b0a393304 service neutron] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:59:13.883039 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01b89500-a22f-4c85-9918-475b0a393304 service neutron] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:59:13.883324 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01b89500-a22f-4c85-9918-475b0a393304 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:59:13.884912 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-01b89500-a22f-4c85-9918-475b0a393304 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:59:13.991925 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01b89500-a22f-4c85-9918-475b0a393304 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:59:13.993574 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01b89500-a22f-4c85-9918-475b0a393304 service neutron] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:59:14.003150 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:59:14.011961 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 658/1334] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 15:59:13 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 311 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 15:59:16.203810 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7378f37f-d60b-445d-a134-6ac1c91036e3 None None] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:59:16.204343 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7378f37f-d60b-445d-a134-6ac1c91036e3 None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:59:16.204343 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7378f37f-d60b-445d-a134-6ac1c91036e3 None None] PATH_INFO: `/` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:59:16.205695 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 677/1335] 10.222.0.50 () {54 vars in 927 bytes} [Tue Sep 13 15:59:16 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:59:16.211465 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5d80fa9-5c9c-48d0-8824-1e4f460beea0 None None] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:59:16.211964 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5d80fa9-5c9c-48d0-8824-1e4f460beea0 None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:59:16.212067 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5d80fa9-5c9c-48d0-8824-1e4f460beea0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:59:16.276455 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.auth.core [None req-d5d80fa9-5c9c-48d0-8824-1e4f460beea0 None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81438) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:59:16.277643 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-d5d80fa9-5c9c-48d0-8824-1e4f460beea0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:59:16.378702 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 659/1336] 10.222.0.50 () {58 vars in 1024 bytes} [Tue Sep 13 15:59:16 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 169 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:59:16.661708 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7378f37f-d60b-445d-a134-6ac1c91036e3 None None] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:59:16.663301 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-7378f37f-d60b-445d-a134-6ac1c91036e3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:59:16.704246 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:59:16.767556 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7378f37f-d60b-445d-a134-6ac1c91036e3 None None] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:59:16.841868 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec63e02d-ccd4-430d-9081-0dd3d31aad00 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f436f07939d4855a21b195bc7566244', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:59:16.842459 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec63e02d-ccd4-430d-9081-0dd3d31aad00 service neutron] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:59:16.842735 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec63e02d-ccd4-430d-9081-0dd3d31aad00 service neutron] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:59:16.843030 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec63e02d-ccd4-430d-9081-0dd3d31aad00 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:59:16.844613 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-ec63e02d-ccd4-430d-9081-0dd3d31aad00 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:59:16.927727 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:59:16.953787 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec63e02d-ccd4-430d-9081-0dd3d31aad00 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:59:16.955266 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec63e02d-ccd4-430d-9081-0dd3d31aad00 service neutron] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:59:16.973811 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 678/1337] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 15:59:16 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 313 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 15:59:19.450644 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9bd1182-0fa9-40a7-a3af-0a7d762b730c None None] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:59:19.451236 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9bd1182-0fa9-40a7-a3af-0a7d762b730c None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:59:19.451582 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9bd1182-0fa9-40a7-a3af-0a7d762b730c None None] PATH_INFO: `/` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:59:19.454045 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 660/1338] 10.222.0.50 () {54 vars in 927 bytes} [Tue Sep 13 15:59:19 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:59:19.459242 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-885f63c1-70a4-49d8-9af6-b783a6e3bbe6 None None] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:59:19.459627 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-885f63c1-70a4-49d8-9af6-b783a6e3bbe6 None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:59:19.459791 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-885f63c1-70a4-49d8-9af6-b783a6e3bbe6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:59:19.524654 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.auth.core [None req-885f63c1-70a4-49d8-9af6-b783a6e3bbe6 None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81437) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:59:19.525849 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-885f63c1-70a4-49d8-9af6-b783a6e3bbe6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:59:19.623893 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 679/1339] 10.222.0.50 () {58 vars in 1024 bytes} [Tue Sep 13 15:59:19 2022] POST /identity/v3/auth/tokens => generated 3238 bytes in 166 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:59:19.903633 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9bd1182-0fa9-40a7-a3af-0a7d762b730c None None] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:59:19.905166 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-f9bd1182-0fa9-40a7-a3af-0a7d762b730c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:59:19.980690 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:59:20.007657 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9bd1182-0fa9-40a7-a3af-0a7d762b730c None None] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:59:20.082299 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99695bfa-5797-4d8e-a6cc-5cea5183f2d5 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f436f07939d4855a21b195bc7566244', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:59:20.082932 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99695bfa-5797-4d8e-a6cc-5cea5183f2d5 service neutron] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:59:20.083215 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99695bfa-5797-4d8e-a6cc-5cea5183f2d5 service neutron] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:59:20.083487 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99695bfa-5797-4d8e-a6cc-5cea5183f2d5 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:59:20.085062 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-99695bfa-5797-4d8e-a6cc-5cea5183f2d5 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:59:20.195000 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99695bfa-5797-4d8e-a6cc-5cea5183f2d5 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:59:20.196374 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99695bfa-5797-4d8e-a6cc-5cea5183f2d5 service neutron] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:59:20.214426 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 661/1340] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 15:59:19 2022] GET /identity/v3/auth/tokens => generated 3238 bytes in 311 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 15:59:24.400935 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e640bba6-07a0-41c5-81f4-226c47b3457c None None] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:59:24.401496 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e640bba6-07a0-41c5-81f4-226c47b3457c None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:59:24.401496 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e640bba6-07a0-41c5-81f4-226c47b3457c None None] PATH_INFO: `/` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:59:24.402817 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 680/1341] 10.222.0.50 () {54 vars in 927 bytes} [Tue Sep 13 15:59:24 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:59:24.408136 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19e90bc7-be2d-44c8-8e81-40a7618ce1f9 None None] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:59:24.408549 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19e90bc7-be2d-44c8-8e81-40a7618ce1f9 None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:59:24.408687 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19e90bc7-be2d-44c8-8e81-40a7618ce1f9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:59:24.414767 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:59:24.477364 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.auth.core [None req-19e90bc7-be2d-44c8-8e81-40a7618ce1f9 None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81438) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:59:24.478501 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-19e90bc7-be2d-44c8-8e81-40a7618ce1f9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:59:24.576396 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 662/1342] 10.222.0.50 () {58 vars in 1024 bytes} [Tue Sep 13 15:59:24 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 169 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:59:24.862343 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e640bba6-07a0-41c5-81f4-226c47b3457c None None] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:59:24.863978 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-e640bba6-07a0-41c5-81f4-226c47b3457c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:59:24.914946 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:59:24.971935 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e640bba6-07a0-41c5-81f4-226c47b3457c None None] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:59:25.046431 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34206668-8e14-4733-93d2-c7c5b04087ab service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f436f07939d4855a21b195bc7566244', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:59:25.047058 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34206668-8e14-4733-93d2-c7c5b04087ab service neutron] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:59:25.047334 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34206668-8e14-4733-93d2-c7c5b04087ab service neutron] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:59:25.047597 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34206668-8e14-4733-93d2-c7c5b04087ab service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:59:25.049168 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-34206668-8e14-4733-93d2-c7c5b04087ab service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:59:25.146498 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:59:25.164188 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34206668-8e14-4733-93d2-c7c5b04087ab service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:59:25.165663 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34206668-8e14-4733-93d2-c7c5b04087ab service neutron] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:59:25.186099 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 681/1343] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 15:59:24 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 324 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 15:59:27.941299 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e72de0c9-b1f4-47dc-a0d7-0e21a9d6d20f None None] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:59:27.941825 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e72de0c9-b1f4-47dc-a0d7-0e21a9d6d20f None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:59:27.941825 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e72de0c9-b1f4-47dc-a0d7-0e21a9d6d20f None None] PATH_INFO: `/` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:59:27.943087 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 663/1344] 10.222.0.50 () {54 vars in 927 bytes} [Tue Sep 13 15:59:27 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:59:27.948426 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e723bce-52e9-437f-87db-4d130a6a1b6b None None] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:59:27.948824 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e723bce-52e9-437f-87db-4d130a6a1b6b None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:59:27.948969 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e723bce-52e9-437f-87db-4d130a6a1b6b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:59:28.014785 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.auth.core [None req-8e723bce-52e9-437f-87db-4d130a6a1b6b None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81437) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:59:28.016010 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-8e723bce-52e9-437f-87db-4d130a6a1b6b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:59:28.117781 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 682/1345] 10.222.0.50 () {58 vars in 1024 bytes} [Tue Sep 13 15:59:27 2022] POST /identity/v3/auth/tokens => generated 3238 bytes in 170 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:59:28.405032 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e72de0c9-b1f4-47dc-a0d7-0e21a9d6d20f None None] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:59:28.406575 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-e72de0c9-b1f4-47dc-a0d7-0e21a9d6d20f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:59:28.497839 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:59:28.517083 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e72de0c9-b1f4-47dc-a0d7-0e21a9d6d20f None None] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:59:28.594748 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bd2ca3e-0433-4461-9fed-87ca65cb068c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f436f07939d4855a21b195bc7566244', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:59:28.595397 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bd2ca3e-0433-4461-9fed-87ca65cb068c service neutron] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:59:28.595673 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bd2ca3e-0433-4461-9fed-87ca65cb068c service neutron] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:59:28.595945 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bd2ca3e-0433-4461-9fed-87ca65cb068c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:59:28.597631 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-3bd2ca3e-0433-4461-9fed-87ca65cb068c service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:59:28.708897 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bd2ca3e-0433-4461-9fed-87ca65cb068c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:59:28.710364 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bd2ca3e-0433-4461-9fed-87ca65cb068c service neutron] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:59:28.728729 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 664/1346] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 15:59:28 2022] GET /identity/v3/auth/tokens => generated 3238 bytes in 324 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 15:59:32.833160 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23426ed3-4872-4298-ba7a-cf92ee82e5a9 None None] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:59:32.833663 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23426ed3-4872-4298-ba7a-cf92ee82e5a9 None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:59:32.833760 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23426ed3-4872-4298-ba7a-cf92ee82e5a9 None None] PATH_INFO: `/` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:59:32.835767 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 683/1347] 10.222.0.50 () {54 vars in 927 bytes} [Tue Sep 13 15:59:32 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:59:32.841286 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06c625d3-5a30-426a-9f03-7da3c61e52d7 None None] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:59:32.841720 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06c625d3-5a30-426a-9f03-7da3c61e52d7 None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:59:32.841808 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06c625d3-5a30-426a-9f03-7da3c61e52d7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:59:32.860662 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:59:32.915468 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.auth.core [None req-06c625d3-5a30-426a-9f03-7da3c61e52d7 None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81438) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:59:32.916692 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-06c625d3-5a30-426a-9f03-7da3c61e52d7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:59:33.043621 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 665/1348] 10.222.0.50 () {58 vars in 1024 bytes} [Tue Sep 13 15:59:32 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 203 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:59:33.322022 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23426ed3-4872-4298-ba7a-cf92ee82e5a9 None None] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:59:33.323748 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-23426ed3-4872-4298-ba7a-cf92ee82e5a9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:59:33.387755 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:59:33.437650 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23426ed3-4872-4298-ba7a-cf92ee82e5a9 None None] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:59:33.512154 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3acd3f9-5400-4e8b-9099-71bbf845e270 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f436f07939d4855a21b195bc7566244', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:59:33.512758 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3acd3f9-5400-4e8b-9099-71bbf845e270 service neutron] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:59:33.513069 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3acd3f9-5400-4e8b-9099-71bbf845e270 service neutron] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:59:33.513342 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3acd3f9-5400-4e8b-9099-71bbf845e270 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:59:33.514988 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-c3acd3f9-5400-4e8b-9099-71bbf845e270 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:59:33.620697 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:59:33.623942 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3acd3f9-5400-4e8b-9099-71bbf845e270 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:59:33.625788 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3acd3f9-5400-4e8b-9099-71bbf845e270 service neutron] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:59:33.643658 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 684/1349] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 15:59:33 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 322 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 15:59:38.484280 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb74c082-1e65-40bd-9de3-b469cc631f69 None None] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:59:38.484810 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb74c082-1e65-40bd-9de3-b469cc631f69 None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:59:38.484810 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb74c082-1e65-40bd-9de3-b469cc631f69 None None] PATH_INFO: `/` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:59:38.486156 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 666/1350] 10.222.0.50 () {54 vars in 921 bytes} [Tue Sep 13 15:59:38 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:59:38.494913 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48b64e3f-6165-4dfc-9ddf-d7af2b79b916 None None] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:59:38.495375 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48b64e3f-6165-4dfc-9ddf-d7af2b79b916 None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:59:38.495702 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48b64e3f-6165-4dfc-9ddf-d7af2b79b916 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:59:38.624019 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.auth.core [None req-48b64e3f-6165-4dfc-9ddf-d7af2b79b916 None None] MFA Rules not processed for user `31480b61418340bd88583f034f67412e`. Rule list: `[]` (Enabled: `True`). {{(pid=81437) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:59:38.625212 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-48b64e3f-6165-4dfc-9ddf-d7af2b79b916 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:59:38.783279 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 685/1351] 10.222.0.50 () {58 vars in 1018 bytes} [Tue Sep 13 15:59:38 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 289 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:59:43.028743 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=59 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:59:43.117651 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72cd4431-0c0a-4706-bc5c-c4285b228a64 None None] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:59:43.118265 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72cd4431-0c0a-4706-bc5c-c4285b228a64 None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:59:43.118611 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72cd4431-0c0a-4706-bc5c-c4285b228a64 None None] PATH_INFO: `/` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:59:43.120854 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 667/1352] 10.222.0.50 () {54 vars in 921 bytes} [Tue Sep 13 15:59:43 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:59:43.125654 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc370533-35eb-4eba-9fac-937445464551 None None] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:59:43.126125 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc370533-35eb-4eba-9fac-937445464551 None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:59:43.126316 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc370533-35eb-4eba-9fac-937445464551 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:59:43.185464 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:59:43.211485 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.auth.core [None req-fc370533-35eb-4eba-9fac-937445464551 None None] MFA Rules not processed for user `31480b61418340bd88583f034f67412e`. Rule list: `[]` (Enabled: `True`). {{(pid=81437) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:59:43.212997 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-fc370533-35eb-4eba-9fac-937445464551 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:59:43.338587 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 686/1353] 10.222.0.50 () {58 vars in 1018 bytes} [Tue Sep 13 15:59:43 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 214 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:59:44.585043 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-629756dd-c579-4e8e-b999-b0489a083fca None None] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:59:44.585573 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-629756dd-c579-4e8e-b999-b0489a083fca None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:59:44.585573 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-629756dd-c579-4e8e-b999-b0489a083fca None None] PATH_INFO: `/` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:59:44.586892 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 668/1354] 10.222.0.50 () {54 vars in 921 bytes} [Tue Sep 13 15:59:44 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:59:44.591871 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b63ca66d-14a6-44a4-9060-cc842a85ec64 None None] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:59:44.592282 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b63ca66d-14a6-44a4-9060-cc842a85ec64 None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:59:44.592419 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b63ca66d-14a6-44a4-9060-cc842a85ec64 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:59:44.668897 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.auth.core [None req-b63ca66d-14a6-44a4-9060-cc842a85ec64 None None] MFA Rules not processed for user `31480b61418340bd88583f034f67412e`. Rule list: `[]` (Enabled: `True`). {{(pid=81437) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:59:44.670091 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-b63ca66d-14a6-44a4-9060-cc842a85ec64 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:59:44.711211 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:59:44.776282 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 687/1355] 10.222.0.50 () {58 vars in 1018 bytes} [Tue Sep 13 15:59:44 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 185 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:59:48.350091 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8997d605-1afc-4946-85ff-e03abfff626d None None] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:59:48.350651 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8997d605-1afc-4946-85ff-e03abfff626d None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:59:48.350651 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8997d605-1afc-4946-85ff-e03abfff626d None None] PATH_INFO: `/` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:59:48.352211 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 669/1356] 10.222.0.50 () {54 vars in 927 bytes} [Tue Sep 13 15:59:48 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:59:48.358050 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b18e4b79-9411-43fc-be4b-4f2f32269424 None None] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:59:48.358515 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b18e4b79-9411-43fc-be4b-4f2f32269424 None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:59:48.358632 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b18e4b79-9411-43fc-be4b-4f2f32269424 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:59:48.434260 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.auth.core [None req-b18e4b79-9411-43fc-be4b-4f2f32269424 None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81437) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:59:48.436114 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-b18e4b79-9411-43fc-be4b-4f2f32269424 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:59:48.542374 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 688/1357] 10.222.0.50 () {58 vars in 1024 bytes} [Tue Sep 13 15:59:48 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 185 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:59:48.855225 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72ac9277-c824-4732-a169-8fb56dc6a85d None None] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:59:48.855648 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72ac9277-c824-4732-a169-8fb56dc6a85d None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:59:48.856033 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72ac9277-c824-4732-a169-8fb56dc6a85d None None] PATH_INFO: `/` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:59:48.857569 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 670/1358] 10.222.0.50 () {54 vars in 962 bytes} [Tue Sep 13 15:59:48 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:59:48.872592 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4057cd0e-48ef-4a2d-8a2e-c459261cda84 None None] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:59:48.872887 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4057cd0e-48ef-4a2d-8a2e-c459261cda84 None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:59:48.873166 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4057cd0e-48ef-4a2d-8a2e-c459261cda84 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:59:48.896403 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:59:48.960604 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.auth.core [None req-4057cd0e-48ef-4a2d-8a2e-c459261cda84 None None] MFA Rules not processed for user `4e039b46c5e3445bbb39904b00c5430e`. Rule list: `[]` (Enabled: `True`). {{(pid=81437) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:59:48.961941 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-4057cd0e-48ef-4a2d-8a2e-c459261cda84 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:59:49.097447 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 689/1359] 10.222.0.50 () {58 vars in 1059 bytes} [Tue Sep 13 15:59:48 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 226 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 13 15:59:49.106249 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72ac9277-c824-4732-a169-8fb56dc6a85d None None] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:59:49.107776 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-72ac9277-c824-4732-a169-8fb56dc6a85d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:59:49.269089 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72ac9277-c824-4732-a169-8fb56dc6a85d None None] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:59:49.351044 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:59:49.393449 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f18ce18a-b9c1-4753-bbdd-b19ac2490455 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4e039b46c5e3445bbb39904b00c5430e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'admin', 'ResellerAdmin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:59:49.394052 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f18ce18a-b9c1-4753-bbdd-b19ac2490455 service nova] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:59:49.394327 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f18ce18a-b9c1-4753-bbdd-b19ac2490455 service nova] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:59:49.394651 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f18ce18a-b9c1-4753-bbdd-b19ac2490455 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:59:49.397434 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-f18ce18a-b9c1-4753-bbdd-b19ac2490455 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:59:49.505621 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f18ce18a-b9c1-4753-bbdd-b19ac2490455 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:59:49.507404 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f18ce18a-b9c1-4753-bbdd-b19ac2490455 service nova] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:59:49.527290 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 671/1360] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 15:59:49 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 421 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 15:59:51.918584 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-800bc27a-a269-4283-bd7b-259db5519428 None None] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:59:51.919381 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-800bc27a-a269-4283-bd7b-259db5519428 None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:59:51.919381 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-800bc27a-a269-4283-bd7b-259db5519428 None None] PATH_INFO: `/` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:59:51.920758 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 690/1361] 10.222.0.50 () {54 vars in 927 bytes} [Tue Sep 13 15:59:51 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:59:51.930675 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-791ef1b0-fbc9-46e3-8418-c86e81a2b1c6 None None] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:59:51.930675 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-791ef1b0-fbc9-46e3-8418-c86e81a2b1c6 None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:59:51.930675 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-791ef1b0-fbc9-46e3-8418-c86e81a2b1c6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:59:52.013389 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.auth.core [None req-791ef1b0-fbc9-46e3-8418-c86e81a2b1c6 None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81438) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:59:52.014661 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-791ef1b0-fbc9-46e3-8418-c86e81a2b1c6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:59:52.061815 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:59:52.125391 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 672/1362] 10.222.0.50 () {58 vars in 1024 bytes} [Tue Sep 13 15:59:51 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 198 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:59:52.418576 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-800bc27a-a269-4283-bd7b-259db5519428 None None] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:59:52.420230 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-800bc27a-a269-4283-bd7b-259db5519428 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:59:52.490525 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:59:52.595220 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-800bc27a-a269-4283-bd7b-259db5519428 None None] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:59:52.721898 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cd8cb63-c790-43f1-b18f-1a2242b183a0 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4e039b46c5e3445bbb39904b00c5430e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'admin', 'ResellerAdmin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:59:52.721898 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cd8cb63-c790-43f1-b18f-1a2242b183a0 service nova] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:59:52.722156 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cd8cb63-c790-43f1-b18f-1a2242b183a0 service nova] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:59:52.722266 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cd8cb63-c790-43f1-b18f-1a2242b183a0 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:59:52.724365 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-5cd8cb63-c790-43f1-b18f-1a2242b183a0 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:59:52.749919 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:59:52.822454 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5cd8cb63-c790-43f1-b18f-1a2242b183a0 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:59:52.824144 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5cd8cb63-c790-43f1-b18f-1a2242b183a0 service nova] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:59:52.842122 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 691/1363] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 15:59:52 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 424 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 15:59:55.384461 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54f9a162-846a-4b26-af6a-5964d5242b2f None None] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:59:55.385013 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54f9a162-846a-4b26-af6a-5964d5242b2f None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:59:55.385013 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54f9a162-846a-4b26-af6a-5964d5242b2f None None] PATH_INFO: `/` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:59:55.386340 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 673/1364] 10.222.0.50 () {54 vars in 927 bytes} [Tue Sep 13 15:59:55 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:59:55.391675 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11d7bc41-fa0b-4253-a2c0-807ad67b2b4c None None] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:59:55.392082 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11d7bc41-fa0b-4253-a2c0-807ad67b2b4c None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:59:55.392258 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11d7bc41-fa0b-4253-a2c0-807ad67b2b4c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:59:55.461964 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.auth.core [None req-11d7bc41-fa0b-4253-a2c0-807ad67b2b4c None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81437) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:59:55.463199 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-11d7bc41-fa0b-4253-a2c0-807ad67b2b4c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:59:55.565199 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:59:55.572697 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 692/1365] 10.222.0.50 () {58 vars in 1024 bytes} [Tue Sep 13 15:59:55 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 181 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:59:55.860351 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d56d24b-c5e6-4bc6-b71b-7535b74ced30 None None] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:59:55.860666 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d56d24b-c5e6-4bc6-b71b-7535b74ced30 None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:59:55.860946 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d56d24b-c5e6-4bc6-b71b-7535b74ced30 None None] PATH_INFO: `/` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:59:55.862392 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 674/1366] 10.222.0.50 () {54 vars in 962 bytes} [Tue Sep 13 15:59:55 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:59:55.868152 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e5321e8-9cdb-4bec-a365-6c67c3920033 None None] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:59:55.868506 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e5321e8-9cdb-4bec-a365-6c67c3920033 None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:59:55.868911 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e5321e8-9cdb-4bec-a365-6c67c3920033 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:59:55.951095 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.auth.core [None req-2e5321e8-9cdb-4bec-a365-6c67c3920033 None None] MFA Rules not processed for user `4e039b46c5e3445bbb39904b00c5430e`. Rule list: `[]` (Enabled: `True`). {{(pid=81437) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:59:55.952306 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-2e5321e8-9cdb-4bec-a365-6c67c3920033 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:59:56.085696 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 693/1367] 10.222.0.50 () {58 vars in 1059 bytes} [Tue Sep 13 15:59:55 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 218 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:59:56.093461 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d56d24b-c5e6-4bc6-b71b-7535b74ced30 None None] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:59:56.095486 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-5d56d24b-c5e6-4bc6-b71b-7535b74ced30 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:59:56.267828 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d56d24b-c5e6-4bc6-b71b-7535b74ced30 None None] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:59:56.290234 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:59:56.392863 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce7b6f27-e473-4835-93c1-619a5fef0178 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4e039b46c5e3445bbb39904b00c5430e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'admin', 'ResellerAdmin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:59:56.393559 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce7b6f27-e473-4835-93c1-619a5fef0178 service nova] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:59:56.393737 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce7b6f27-e473-4835-93c1-619a5fef0178 service nova] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:59:56.394047 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce7b6f27-e473-4835-93c1-619a5fef0178 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:59:56.395800 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-ce7b6f27-e473-4835-93c1-619a5fef0178 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:59:56.497252 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce7b6f27-e473-4835-93c1-619a5fef0178 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:59:56.498876 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce7b6f27-e473-4835-93c1-619a5fef0178 service nova] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:59:56.517575 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 675/1368] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 15:59:56 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 424 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 15:59:56.556436 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98dd512c-5086-4aa8-b7e2-06ab023329b0 None None] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:59:56.556991 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98dd512c-5086-4aa8-b7e2-06ab023329b0 None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:59:56.556991 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98dd512c-5086-4aa8-b7e2-06ab023329b0 None None] PATH_INFO: `/` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:59:56.559246 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 694/1369] 10.222.0.50 () {54 vars in 927 bytes} [Tue Sep 13 15:59:56 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:59:56.563706 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db702e63-1650-4523-95cf-9e0f47ffddbf None None] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:59:56.564001 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db702e63-1650-4523-95cf-9e0f47ffddbf None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:59:56.564275 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db702e63-1650-4523-95cf-9e0f47ffddbf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:59:56.599742 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:59:56.644544 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.auth.core [None req-db702e63-1650-4523-95cf-9e0f47ffddbf None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81438) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:59:56.645984 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-db702e63-1650-4523-95cf-9e0f47ffddbf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:59:56.768160 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 676/1370] 10.222.0.50 () {58 vars in 1024 bytes} [Tue Sep 13 15:59:56 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 205 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:59:57.098713 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b18b4153-c893-4d60-9787-ed7a2e6c533d None None] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:59:57.098713 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b18b4153-c893-4d60-9787-ed7a2e6c533d None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:59:57.098713 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b18b4153-c893-4d60-9787-ed7a2e6c533d None None] PATH_INFO: `/` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:59:57.100421 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 695/1371] 10.222.0.50 () {54 vars in 964 bytes} [Tue Sep 13 15:59:57 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:59:57.104862 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e8f4cd4-7823-4b04-b651-5e2f3de9401a None None] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:59:57.105145 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e8f4cd4-7823-4b04-b651-5e2f3de9401a None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:59:57.105417 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e8f4cd4-7823-4b04-b651-5e2f3de9401a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:59:57.185640 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.auth.core [None req-6e8f4cd4-7823-4b04-b651-5e2f3de9401a None None] MFA Rules not processed for user `7fd6b5afd67b48f9b0f0597ccd4cf4df`. Rule list: `[]` (Enabled: `True`). {{(pid=81438) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:59:57.186931 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-6e8f4cd4-7823-4b04-b651-5e2f3de9401a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:59:57.238849 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:59:57.269345 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 677/1372] 10.222.0.50 () {58 vars in 1061 bytes} [Tue Sep 13 15:59:57 2022] POST /identity/v3/auth/tokens => generated 3120 bytes in 165 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:59:57.276798 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b18b4153-c893-4d60-9787-ed7a2e6c533d None None] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:59:57.278323 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-b18b4153-c893-4d60-9787-ed7a2e6c533d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:59:57.333675 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:59:57.392263 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b18b4153-c893-4d60-9787-ed7a2e6c533d None None] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:59:57.477976 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-587e0276-f674-4e63-8708-2bb419078f3c service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7fd6b5afd67b48f9b0f0597ccd4cf4df', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:59:57.478589 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-587e0276-f674-4e63-8708-2bb419078f3c service cinder] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:59:57.478871 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-587e0276-f674-4e63-8708-2bb419078f3c service cinder] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:59:57.479178 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-587e0276-f674-4e63-8708-2bb419078f3c service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:59:57.480798 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-587e0276-f674-4e63-8708-2bb419078f3c service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:59:57.596168 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:59:57.609917 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-587e0276-f674-4e63-8708-2bb419078f3c service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:59:57.611458 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-587e0276-f674-4e63-8708-2bb419078f3c service cinder] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:59:57.631963 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 696/1373] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 15:59:57 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 355 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 15:59:58.791038 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3f303bd-3379-4fbd-a0cc-8bbebac33ef9 None None] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:59:58.791890 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3f303bd-3379-4fbd-a0cc-8bbebac33ef9 None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:59:58.791890 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3f303bd-3379-4fbd-a0cc-8bbebac33ef9 None None] PATH_INFO: `/` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:59:58.793177 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 678/1374] 10.222.0.50 () {54 vars in 927 bytes} [Tue Sep 13 15:59:58 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:59:58.798623 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5528c628-0eb1-4cb0-ab77-2d1ce73411b8 None None] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:59:58.799084 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5528c628-0eb1-4cb0-ab77-2d1ce73411b8 None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:59:58.799326 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5528c628-0eb1-4cb0-ab77-2d1ce73411b8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:59:58.869276 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.auth.core [None req-5528c628-0eb1-4cb0-ab77-2d1ce73411b8 None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81437) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:59:58.870538 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-5528c628-0eb1-4cb0-ab77-2d1ce73411b8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:59:58.972672 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 697/1375] 10.222.0.50 () {58 vars in 1024 bytes} [Tue Sep 13 15:59:58 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 175 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:59:59.269938 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3f303bd-3379-4fbd-a0cc-8bbebac33ef9 None None] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:59:59.271670 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-d3f303bd-3379-4fbd-a0cc-8bbebac33ef9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:59:59.429391 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3f303bd-3379-4fbd-a0cc-8bbebac33ef9 None None] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:59:59.486530 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:59:59.553666 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-151b854f-d2e7-46be-8420-88a3a92af6f5 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4e039b46c5e3445bbb39904b00c5430e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'admin', 'ResellerAdmin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:59:59.553976 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-151b854f-d2e7-46be-8420-88a3a92af6f5 service nova] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:59:59.554247 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-151b854f-d2e7-46be-8420-88a3a92af6f5 service nova] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:59:59.554511 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-151b854f-d2e7-46be-8420-88a3a92af6f5 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:59:59.556405 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-151b854f-d2e7-46be-8420-88a3a92af6f5 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:59:59.652664 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-151b854f-d2e7-46be-8420-88a3a92af6f5 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:59:59.654255 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-151b854f-d2e7-46be-8420-88a3a92af6f5 service nova] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:59:59.674815 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 679/1376] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 15:59:59 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 406 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 16:00:01.919698 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16b57570-d2d0-4a19-9f2f-43ab357ce6af None None] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:00:01.920259 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16b57570-d2d0-4a19-9f2f-43ab357ce6af None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:00:01.920259 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16b57570-d2d0-4a19-9f2f-43ab357ce6af None None] PATH_INFO: `/` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:00:01.921657 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 698/1377] 10.222.0.50 () {54 vars in 927 bytes} [Tue Sep 13 16:00:01 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 16:00:01.927228 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d46f380f-afd4-4652-9d6f-9fc4f09e71f6 None None] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:00:01.927682 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d46f380f-afd4-4652-9d6f-9fc4f09e71f6 None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:00:01.927980 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d46f380f-afd4-4652-9d6f-9fc4f09e71f6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:00:01.995295 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.auth.core [None req-d46f380f-afd4-4652-9d6f-9fc4f09e71f6 None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81438) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:00:01.996638 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-d46f380f-afd4-4652-9d6f-9fc4f09e71f6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:00:02.004404 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:00:02.115044 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 680/1378] 10.222.0.50 () {58 vars in 1024 bytes} [Tue Sep 13 16:00:01 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 189 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:00:02.399281 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16b57570-d2d0-4a19-9f2f-43ab357ce6af None None] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:00:02.400963 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-16b57570-d2d0-4a19-9f2f-43ab357ce6af None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:00:02.465791 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:00:02.556161 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16b57570-d2d0-4a19-9f2f-43ab357ce6af None None] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:00:02.679830 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb95512c-000a-4c37-a5b8-3a7540564447 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4e039b46c5e3445bbb39904b00c5430e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'admin', 'ResellerAdmin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:00:02.680539 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb95512c-000a-4c37-a5b8-3a7540564447 service nova] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:00:02.680915 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb95512c-000a-4c37-a5b8-3a7540564447 service nova] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:00:02.681291 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb95512c-000a-4c37-a5b8-3a7540564447 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:00:02.683033 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-fb95512c-000a-4c37-a5b8-3a7540564447 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:00:02.705378 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:00:02.796771 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb95512c-000a-4c37-a5b8-3a7540564447 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:00:02.798576 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb95512c-000a-4c37-a5b8-3a7540564447 service nova] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:00:02.818197 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 699/1379] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 16:00:02 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 419 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 16:00:05.052588 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b47f858-c1f7-49f4-a7db-63c00afa3adc None None] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:00:05.053928 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b47f858-c1f7-49f4-a7db-63c00afa3adc None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:00:05.053928 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b47f858-c1f7-49f4-a7db-63c00afa3adc None None] PATH_INFO: `/` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:00:05.054426 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 681/1380] 10.222.0.50 () {54 vars in 927 bytes} [Tue Sep 13 16:00:05 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 16:00:05.061214 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-baaa9710-fcf1-4971-9277-886967e968d2 None None] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:00:05.061214 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-baaa9710-fcf1-4971-9277-886967e968d2 None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:00:05.061214 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-baaa9710-fcf1-4971-9277-886967e968d2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:00:05.137284 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.auth.core [None req-baaa9710-fcf1-4971-9277-886967e968d2 None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81437) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:00:05.138592 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-baaa9710-fcf1-4971-9277-886967e968d2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:00:05.233910 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:00:05.239930 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 700/1381] 10.222.0.50 () {58 vars in 1024 bytes} [Tue Sep 13 16:00:05 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 181 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:00:05.537745 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b47f858-c1f7-49f4-a7db-63c00afa3adc None None] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:00:05.539388 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-9b47f858-c1f7-49f4-a7db-63c00afa3adc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:00:05.680861 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:00:05.691326 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b47f858-c1f7-49f4-a7db-63c00afa3adc None None] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:00:05.808384 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b61cda6-6329-4ff3-858b-5f5620b9b0b6 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4e039b46c5e3445bbb39904b00c5430e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'admin', 'ResellerAdmin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:00:05.808923 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b61cda6-6329-4ff3-858b-5f5620b9b0b6 service nova] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:00:05.809199 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b61cda6-6329-4ff3-858b-5f5620b9b0b6 service nova] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:00:05.809479 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b61cda6-6329-4ff3-858b-5f5620b9b0b6 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:00:05.811202 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-2b61cda6-6329-4ff3-858b-5f5620b9b0b6 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:00:05.903761 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b61cda6-6329-4ff3-858b-5f5620b9b0b6 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:00:05.905798 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b61cda6-6329-4ff3-858b-5f5620b9b0b6 service nova] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:00:05.918651 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:00:05.928620 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 682/1382] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 16:00:05 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 391 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 13 16:00:08.165415 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e72f3594-045f-4418-b290-843a134afb67 None None] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:00:08.166368 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e72f3594-045f-4418-b290-843a134afb67 None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:00:08.166368 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e72f3594-045f-4418-b290-843a134afb67 None None] PATH_INFO: `/` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:00:08.167502 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 701/1383] 10.222.0.50 () {54 vars in 927 bytes} [Tue Sep 13 16:00:08 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 16:00:08.173247 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22d5fd7d-3830-4a44-966d-deba76f87928 None None] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:00:08.173693 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22d5fd7d-3830-4a44-966d-deba76f87928 None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:00:08.173934 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22d5fd7d-3830-4a44-966d-deba76f87928 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:00:08.253067 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.auth.core [None req-22d5fd7d-3830-4a44-966d-deba76f87928 None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81438) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:00:08.254696 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-22d5fd7d-3830-4a44-966d-deba76f87928 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:00:08.367948 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 683/1384] 10.222.0.50 () {58 vars in 1024 bytes} [Tue Sep 13 16:00:08 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 196 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:00:08.681493 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e72f3594-045f-4418-b290-843a134afb67 None None] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:00:08.683092 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-e72f3594-045f-4418-b290-843a134afb67 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:00:08.839729 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e72f3594-045f-4418-b290-843a134afb67 None None] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:00:08.919239 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:00:09.093162 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fae740c4-a407-4acc-a225-d94bc1e9102b service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4e039b46c5e3445bbb39904b00c5430e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'admin', 'ResellerAdmin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:00:09.093780 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fae740c4-a407-4acc-a225-d94bc1e9102b service nova] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:00:09.094066 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fae740c4-a407-4acc-a225-d94bc1e9102b service nova] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:00:09.094355 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fae740c4-a407-4acc-a225-d94bc1e9102b service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:00:09.096645 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-fae740c4-a407-4acc-a225-d94bc1e9102b service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:00:09.188098 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fae740c4-a407-4acc-a225-d94bc1e9102b service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:00:09.189750 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fae740c4-a407-4acc-a225-d94bc1e9102b service nova] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:00:09.207635 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 702/1385] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 16:00:08 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 527 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 16:00:11.510479 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bc44b07-c2f6-4bdd-92a1-a13bf95fc8d6 None None] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:00:11.511200 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bc44b07-c2f6-4bdd-92a1-a13bf95fc8d6 None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:00:11.511200 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bc44b07-c2f6-4bdd-92a1-a13bf95fc8d6 None None] PATH_INFO: `/` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:00:11.512608 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 684/1386] 10.222.0.50 () {54 vars in 927 bytes} [Tue Sep 13 16:00:11 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 13 16:00:11.518770 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd8a85cd-7360-4590-8d8c-d06835e12d76 None None] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:00:11.519442 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd8a85cd-7360-4590-8d8c-d06835e12d76 None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:00:11.519442 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd8a85cd-7360-4590-8d8c-d06835e12d76 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:00:11.591345 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.auth.core [None req-cd8a85cd-7360-4590-8d8c-d06835e12d76 None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81437) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:00:11.592920 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-cd8a85cd-7360-4590-8d8c-d06835e12d76 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:00:11.634746 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:00:11.706821 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 703/1387] 10.222.0.50 () {58 vars in 1024 bytes} [Tue Sep 13 16:00:11 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 190 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:00:12.004536 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bc44b07-c2f6-4bdd-92a1-a13bf95fc8d6 None None] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:00:12.006058 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-8bc44b07-c2f6-4bdd-92a1-a13bf95fc8d6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:00:12.088859 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:00:12.161999 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bc44b07-c2f6-4bdd-92a1-a13bf95fc8d6 None None] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:00:12.276889 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a03d30b-01b4-4a00-8bae-c0b63b09dad3 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4e039b46c5e3445bbb39904b00c5430e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'admin', 'ResellerAdmin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:00:12.277415 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a03d30b-01b4-4a00-8bae-c0b63b09dad3 service nova] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:00:12.277695 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a03d30b-01b4-4a00-8bae-c0b63b09dad3 service nova] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:00:12.277968 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a03d30b-01b4-4a00-8bae-c0b63b09dad3 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:00:12.279623 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-7a03d30b-01b4-4a00-8bae-c0b63b09dad3 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:00:12.319067 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:00:12.372717 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a03d30b-01b4-4a00-8bae-c0b63b09dad3 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:00:12.374326 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a03d30b-01b4-4a00-8bae-c0b63b09dad3 service nova] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:00:12.393993 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 685/1388] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 16:00:12 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 390 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 16:00:14.160140 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-beead609-6157-429a-acbf-a78850ca102c None None] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:00:14.160699 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-beead609-6157-429a-acbf-a78850ca102c None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:00:14.160699 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-beead609-6157-429a-acbf-a78850ca102c None None] PATH_INFO: `/` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:00:14.162081 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 704/1389] 10.222.0.50 () {54 vars in 927 bytes} [Tue Sep 13 16:00:14 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 16:00:14.167833 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b935bb5b-645d-4102-84e3-163b53b2de8f None None] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:00:14.168233 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b935bb5b-645d-4102-84e3-163b53b2de8f None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:00:14.168521 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b935bb5b-645d-4102-84e3-163b53b2de8f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:00:14.235473 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.auth.core [None req-b935bb5b-645d-4102-84e3-163b53b2de8f None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81438) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:00:14.237326 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-b935bb5b-645d-4102-84e3-163b53b2de8f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:00:14.341850 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 686/1390] 10.222.0.50 () {58 vars in 1024 bytes} [Tue Sep 13 16:00:14 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 175 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:00:14.615336 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2d70aed-68b9-48eb-b42f-e792d9467da3 None None] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:00:14.615542 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2d70aed-68b9-48eb-b42f-e792d9467da3 None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:00:14.615794 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2d70aed-68b9-48eb-b42f-e792d9467da3 None None] PATH_INFO: `/` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:00:14.617131 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 705/1391] 10.222.0.50 () {54 vars in 927 bytes} [Tue Sep 13 16:00:14 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 16:00:14.622460 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e382da7-f122-471f-88f1-3fb2171ce9e2 None None] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:00:14.622850 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e382da7-f122-471f-88f1-3fb2171ce9e2 None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:00:14.623534 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e382da7-f122-471f-88f1-3fb2171ce9e2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:00:14.639427 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:00:14.651954 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f662bf7-836e-4f51-b3a0-02a3a3cc95d4 None None] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:00:14.652588 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f662bf7-836e-4f51-b3a0-02a3a3cc95d4 None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:00:14.652862 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f662bf7-836e-4f51-b3a0-02a3a3cc95d4 None None] PATH_INFO: `/` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:00:14.655165 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 706/1392] 10.222.0.50 () {54 vars in 966 bytes} [Tue Sep 13 16:00:14 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 13 16:00:14.661334 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-296975b4-2c84-4d89-aad8-9f26632682d8 None None] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:00:14.661616 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-296975b4-2c84-4d89-aad8-9f26632682d8 None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:00:14.661892 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-296975b4-2c84-4d89-aad8-9f26632682d8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:00:14.693357 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.auth.core [None req-5e382da7-f122-471f-88f1-3fb2171ce9e2 None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81438) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:00:14.694602 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-5e382da7-f122-471f-88f1-3fb2171ce9e2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:00:14.737318 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.auth.core [None req-296975b4-2c84-4d89-aad8-9f26632682d8 None None] MFA Rules not processed for user `6ab5f8018f464be69d4af7683f553738`. Rule list: `[]` (Enabled: `True`). {{(pid=81437) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:00:14.738700 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-296975b4-2c84-4d89-aad8-9f26632682d8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:00:14.796435 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 687/1393] 10.222.0.50 () {58 vars in 1024 bytes} [Tue Sep 13 16:00:14 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 175 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:00:14.817663 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 707/1394] 10.222.0.50 () {58 vars in 1063 bytes} [Tue Sep 13 16:00:14 2022] POST /identity/v3/auth/tokens => generated 3120 bytes in 157 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 13 16:00:14.826774 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e382da7-f122-471f-88f1-3fb2171ce9e2 None None] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:00:14.828120 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-5e382da7-f122-471f-88f1-3fb2171ce9e2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:00:14.920039 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e382da7-f122-471f-88f1-3fb2171ce9e2 None None] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:00:14.928070 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:00:14.997976 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fccce0d-c844-4b3c-8e0f-bdc476050e2f service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6ab5f8018f464be69d4af7683f553738', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:00:14.998580 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fccce0d-c844-4b3c-8e0f-bdc476050e2f service glance] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:00:14.998856 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fccce0d-c844-4b3c-8e0f-bdc476050e2f service glance] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:00:14.999152 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fccce0d-c844-4b3c-8e0f-bdc476050e2f service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:00:15.000712 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-1fccce0d-c844-4b3c-8e0f-bdc476050e2f service glance] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:00:15.091802 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-296975b4-2c84-4d89-aad8-9f26632682d8 None None] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:00:15.092992 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-296975b4-2c84-4d89-aad8-9f26632682d8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:00:15.125605 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:00:15.127890 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fccce0d-c844-4b3c-8e0f-bdc476050e2f service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:00:15.129342 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fccce0d-c844-4b3c-8e0f-bdc476050e2f service glance] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:00:15.149273 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 688/1395] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 16:00:14 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 323 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 16:00:15.219341 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-296975b4-2c84-4d89-aad8-9f26632682d8 None None] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:00:15.344057 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8499511-89a6-45df-b780-34c7bc2a151f service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4e039b46c5e3445bbb39904b00c5430e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'admin', 'ResellerAdmin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:00:15.344682 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8499511-89a6-45df-b780-34c7bc2a151f service nova] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:00:15.344955 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8499511-89a6-45df-b780-34c7bc2a151f service nova] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:00:15.345291 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8499511-89a6-45df-b780-34c7bc2a151f service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:00:15.347676 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-d8499511-89a6-45df-b780-34c7bc2a151f service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:00:15.370472 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:00:15.453065 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8499511-89a6-45df-b780-34c7bc2a151f service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:00:15.454726 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8499511-89a6-45df-b780-34c7bc2a151f service nova] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:00:15.473398 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 708/1396] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 16:00:15 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 382 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 16:00:15.922313 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-360585ab-a152-4d98-92a4-576a52f2794a None None] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:00:15.922822 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-360585ab-a152-4d98-92a4-576a52f2794a None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:00:15.923066 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-360585ab-a152-4d98-92a4-576a52f2794a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:00:15.971585 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:00:15.994151 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.auth.core [None req-360585ab-a152-4d98-92a4-576a52f2794a None None] MFA Rules not processed for user `f8364b1a190d4218bf353a89bdb81448`. Rule list: `[]` (Enabled: `True`). {{(pid=81438) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:00:15.995395 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-360585ab-a152-4d98-92a4-576a52f2794a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:00:16.087925 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 689/1397] 10.222.0.50 () {58 vars in 1009 bytes} [Tue Sep 13 16:00:15 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 167 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:00:16.135188 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1774427d-acf3-482a-8212-51a11c1b6f7b None None] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:00:16.135478 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1774427d-acf3-482a-8212-51a11c1b6f7b None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:00:16.135758 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1774427d-acf3-482a-8212-51a11c1b6f7b None None] PATH_INFO: `/` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:00:16.137119 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 709/1398] 10.222.0.50 () {54 vars in 943 bytes} [Tue Sep 13 16:00:16 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 16:00:16.143811 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ba982af-043a-4cd0-b8c5-2677ea649bd1 None None] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:00:16.144136 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ba982af-043a-4cd0-b8c5-2677ea649bd1 None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:00:16.144416 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ba982af-043a-4cd0-b8c5-2677ea649bd1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:00:16.221573 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.auth.core [None req-5ba982af-043a-4cd0-b8c5-2677ea649bd1 None None] MFA Rules not processed for user `b00ab0d6321d483188bc69deaddcdbbc`. Rule list: `[]` (Enabled: `True`). {{(pid=81438) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:00:16.222795 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-5ba982af-043a-4cd0-b8c5-2677ea649bd1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:00:16.286205 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:00:16.334935 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 690/1399] 10.222.0.50 () {58 vars in 1040 bytes} [Tue Sep 13 16:00:16 2022] POST /identity/v3/auth/tokens => generated 3304 bytes in 192 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:00:16.344207 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1774427d-acf3-482a-8212-51a11c1b6f7b None None] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:00:16.352750 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-1774427d-acf3-482a-8212-51a11c1b6f7b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:00:16.494448 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1774427d-acf3-482a-8212-51a11c1b6f7b None None] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:00:16.504039 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:00:16.598470 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56e4d605-114a-4aa8-b246-dd582491dfc6 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b00ab0d6321d483188bc69deaddcdbbc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:00:16.599019 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56e4d605-114a-4aa8-b246-dd582491dfc6 service swift] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:00:16.599304 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56e4d605-114a-4aa8-b246-dd582491dfc6 service swift] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:00:16.599575 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56e4d605-114a-4aa8-b246-dd582491dfc6 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:00:16.601196 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-56e4d605-114a-4aa8-b246-dd582491dfc6 service swift] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:00:16.683576 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56e4d605-114a-4aa8-b246-dd582491dfc6 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:00:16.685157 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56e4d605-114a-4aa8-b246-dd582491dfc6 service swift] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:00:16.693721 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 710/1400] 10.222.0.50 () {60 vars in 1384 bytes} [Tue Sep 13 16:00:16 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 350 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Sep 13 16:00:17.686001 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7cd4ac5-10f7-4d0d-8261-524d21f02d81 None None] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:00:17.686547 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7cd4ac5-10f7-4d0d-8261-524d21f02d81 None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:00:17.686547 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7cd4ac5-10f7-4d0d-8261-524d21f02d81 None None] PATH_INFO: `/` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:00:17.687894 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 691/1401] 10.222.0.50 () {54 vars in 927 bytes} [Tue Sep 13 16:00:17 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 16:00:17.696512 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6de7908d-4bdc-49fc-a31d-315aed597de8 None None] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:00:17.696969 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6de7908d-4bdc-49fc-a31d-315aed597de8 None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:00:17.697192 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6de7908d-4bdc-49fc-a31d-315aed597de8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:00:17.761322 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:00:17.782082 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.auth.core [None req-6de7908d-4bdc-49fc-a31d-315aed597de8 None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81437) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:00:17.784343 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-6de7908d-4bdc-49fc-a31d-315aed597de8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:00:17.897246 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 711/1402] 10.222.0.50 () {58 vars in 1024 bytes} [Tue Sep 13 16:00:17 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 202 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:00:18.191385 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7cd4ac5-10f7-4d0d-8261-524d21f02d81 None None] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:00:18.194921 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-d7cd4ac5-10f7-4d0d-8261-524d21f02d81 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:00:18.370933 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7cd4ac5-10f7-4d0d-8261-524d21f02d81 None None] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:00:18.432083 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:00:18.517443 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2aee0f87-3c24-4d78-a344-3a67e8a1fb1f service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4e039b46c5e3445bbb39904b00c5430e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'admin', 'ResellerAdmin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:00:18.518057 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2aee0f87-3c24-4d78-a344-3a67e8a1fb1f service nova] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:00:18.518328 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2aee0f87-3c24-4d78-a344-3a67e8a1fb1f service nova] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:00:18.518597 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2aee0f87-3c24-4d78-a344-3a67e8a1fb1f service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:00:18.521095 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-2aee0f87-3c24-4d78-a344-3a67e8a1fb1f service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:00:18.625336 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2aee0f87-3c24-4d78-a344-3a67e8a1fb1f service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:00:18.627709 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2aee0f87-3c24-4d78-a344-3a67e8a1fb1f service nova] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:00:18.648488 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 692/1403] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 16:00:18 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 457 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 16:00:20.952851 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9620119e-fb2c-4a52-81a2-79b3e03fc4a3 None None] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:00:20.953501 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9620119e-fb2c-4a52-81a2-79b3e03fc4a3 None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:00:20.953501 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9620119e-fb2c-4a52-81a2-79b3e03fc4a3 None None] PATH_INFO: `/` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:00:20.956563 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 712/1404] 10.222.0.50 () {54 vars in 927 bytes} [Tue Sep 13 16:00:20 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 16:00:20.961093 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4384b33e-8f03-42d4-9c1c-2882d58c9745 None None] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:00:20.961559 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4384b33e-8f03-42d4-9c1c-2882d58c9745 None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:00:20.961671 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4384b33e-8f03-42d4-9c1c-2882d58c9745 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:00:21.028207 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:00:21.045042 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.auth.core [None req-4384b33e-8f03-42d4-9c1c-2882d58c9745 None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81438) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:00:21.046369 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-4384b33e-8f03-42d4-9c1c-2882d58c9745 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:00:21.184201 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 693/1405] 10.222.0.50 () {58 vars in 1024 bytes} [Tue Sep 13 16:00:20 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 224 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:00:21.498582 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9620119e-fb2c-4a52-81a2-79b3e03fc4a3 None None] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:00:21.500142 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-9620119e-fb2c-4a52-81a2-79b3e03fc4a3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:00:21.669720 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:00:21.715386 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9620119e-fb2c-4a52-81a2-79b3e03fc4a3 None None] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:00:21.874596 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef72882f-42a2-426c-8812-7ebc192e7d4f service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4e039b46c5e3445bbb39904b00c5430e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'admin', 'ResellerAdmin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:00:21.875231 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef72882f-42a2-426c-8812-7ebc192e7d4f service nova] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:00:21.875506 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef72882f-42a2-426c-8812-7ebc192e7d4f service nova] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:00:21.876322 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef72882f-42a2-426c-8812-7ebc192e7d4f service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:00:21.878145 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-ef72882f-42a2-426c-8812-7ebc192e7d4f service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:00:21.967281 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:00:21.978752 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef72882f-42a2-426c-8812-7ebc192e7d4f service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:00:21.980371 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef72882f-42a2-426c-8812-7ebc192e7d4f service nova] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:00:22.003017 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 713/1406] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 16:00:21 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 505 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 16:00:24.230219 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f53dc82-0742-40b2-b3f8-524f7d93791d None None] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:00:24.230786 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f53dc82-0742-40b2-b3f8-524f7d93791d None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:00:24.230786 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f53dc82-0742-40b2-b3f8-524f7d93791d None None] PATH_INFO: `/` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:00:24.232728 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 694/1407] 10.222.0.50 () {54 vars in 927 bytes} [Tue Sep 13 16:00:24 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 16:00:24.238989 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c10398e-3549-4400-bbd7-395315d0e9f6 None None] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:00:24.239427 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c10398e-3549-4400-bbd7-395315d0e9f6 None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:00:24.239528 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c10398e-3549-4400-bbd7-395315d0e9f6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:00:24.320758 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.auth.core [None req-3c10398e-3549-4400-bbd7-395315d0e9f6 None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81437) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:00:24.321891 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-3c10398e-3549-4400-bbd7-395315d0e9f6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:00:24.430768 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 714/1408] 10.222.0.50 () {58 vars in 1024 bytes} [Tue Sep 13 16:00:24 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 193 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:00:24.725564 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f53dc82-0742-40b2-b3f8-524f7d93791d None None] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:00:24.727126 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-1f53dc82-0742-40b2-b3f8-524f7d93791d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:00:24.869312 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:00:24.898422 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f53dc82-0742-40b2-b3f8-524f7d93791d None None] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:00:25.031307 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9e8b78b-1e72-4b20-bd28-d2d7b95ed1e5 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4e039b46c5e3445bbb39904b00c5430e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'admin', 'ResellerAdmin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:00:25.031955 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9e8b78b-1e72-4b20-bd28-d2d7b95ed1e5 service nova] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:00:25.032237 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9e8b78b-1e72-4b20-bd28-d2d7b95ed1e5 service nova] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:00:25.032517 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9e8b78b-1e72-4b20-bd28-d2d7b95ed1e5 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:00:25.034107 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-f9e8b78b-1e72-4b20-bd28-d2d7b95ed1e5 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:00:25.137314 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:00:25.143314 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9e8b78b-1e72-4b20-bd28-d2d7b95ed1e5 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:00:25.145032 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9e8b78b-1e72-4b20-bd28-d2d7b95ed1e5 service nova] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:00:25.175300 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 695/1409] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 16:00:24 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 450 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 16:00:34.429404 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=74 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:00:35.161142 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=7 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:00:39.886745 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8f58cdf-a3aa-4acd-a392-c690387f49e3 None None] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:00:39.887360 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8f58cdf-a3aa-4acd-a392-c690387f49e3 None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:00:39.887360 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8f58cdf-a3aa-4acd-a392-c690387f49e3 None None] PATH_INFO: `/` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:00:39.888689 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 715/1410] 10.222.0.50 () {54 vars in 918 bytes} [Tue Sep 13 16:00:39 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 16:00:39.893870 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca623db0-e908-4871-890d-65237cff40e2 None None] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:00:39.894285 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca623db0-e908-4871-890d-65237cff40e2 None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:00:39.894426 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca623db0-e908-4871-890d-65237cff40e2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:00:39.970746 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.auth.core [None req-ca623db0-e908-4871-890d-65237cff40e2 None None] MFA Rules not processed for user `31480b61418340bd88583f034f67412e`. Rule list: `[]` (Enabled: `True`). {{(pid=81438) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:00:39.972000 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-ca623db0-e908-4871-890d-65237cff40e2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:00:40.078327 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 696/1411] 10.222.0.50 () {58 vars in 1015 bytes} [Tue Sep 13 16:00:39 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 185 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:00:40.092991 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56b90d8c-fd21-4c89-a49c-8239c1b9d65f None None] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:00:40.093352 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56b90d8c-fd21-4c89-a49c-8239c1b9d65f None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:00:40.093821 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56b90d8c-fd21-4c89-a49c-8239c1b9d65f None None] PATH_INFO: `/` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:00:40.094944 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 716/1412] 10.222.0.50 () {54 vars in 918 bytes} [Tue Sep 13 16:00:40 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 16:00:40.099544 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4390ea1e-4068-4d26-ab4e-5d7aecd7a853 None None] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:00:40.099830 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4390ea1e-4068-4d26-ab4e-5d7aecd7a853 None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:00:40.100109 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4390ea1e-4068-4d26-ab4e-5d7aecd7a853 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:00:40.175150 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:00:40.178354 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.auth.core [None req-4390ea1e-4068-4d26-ab4e-5d7aecd7a853 None None] MFA Rules not processed for user `4e039b46c5e3445bbb39904b00c5430e`. Rule list: `[]` (Enabled: `True`). {{(pid=81438) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:00:40.180267 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-4390ea1e-4068-4d26-ab4e-5d7aecd7a853 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:00:40.296333 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 697/1413] 10.222.0.50 () {58 vars in 1015 bytes} [Tue Sep 13 16:00:40 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 198 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:00:42.877285 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0e3ce87-8171-48c5-b515-059e5c224f9c None None] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:00:42.877833 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0e3ce87-8171-48c5-b515-059e5c224f9c None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:00:42.877833 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0e3ce87-8171-48c5-b515-059e5c224f9c None None] PATH_INFO: `/` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:00:42.879188 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 717/1414] 127.0.0.1 () {54 vars in 917 bytes} [Tue Sep 13 16:00:42 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 16:00:42.884442 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f303cf46-3e28-4f34-8d6d-991ef5565ef4 None None] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:00:42.884894 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f303cf46-3e28-4f34-8d6d-991ef5565ef4 None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:00:42.885004 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f303cf46-3e28-4f34-8d6d-991ef5565ef4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:00:42.951398 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.auth.core [None req-f303cf46-3e28-4f34-8d6d-991ef5565ef4 None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81438) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:00:42.953264 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-f303cf46-3e28-4f34-8d6d-991ef5565ef4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:00:43.007907 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:00:43.050153 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 698/1415] 127.0.0.1 () {58 vars in 1014 bytes} [Tue Sep 13 16:00:42 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 167 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:00:43.338506 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0e3ce87-8171-48c5-b515-059e5c224f9c None None] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:00:43.340099 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-e0e3ce87-8171-48c5-b515-059e5c224f9c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:00:43.483995 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0e3ce87-8171-48c5-b515-059e5c224f9c None None] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:00:43.563037 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:00:43.602842 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1c7a5f1-a461-4bfb-b5fd-0b825946a558 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4e039b46c5e3445bbb39904b00c5430e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'admin', 'ResellerAdmin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:00:43.603400 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1c7a5f1-a461-4bfb-b5fd-0b825946a558 service nova] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:00:43.603677 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1c7a5f1-a461-4bfb-b5fd-0b825946a558 service nova] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:00:43.603947 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1c7a5f1-a461-4bfb-b5fd-0b825946a558 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:00:43.605531 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-c1c7a5f1-a461-4bfb-b5fd-0b825946a558 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:00:43.693313 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1c7a5f1-a461-4bfb-b5fd-0b825946a558 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:00:43.694893 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1c7a5f1-a461-4bfb-b5fd-0b825946a558 service nova] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:00:43.712260 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 718/1416] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 16:00:43 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 374 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 16:00:44.855783 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cee59b6e-b780-4849-ab29-fc1c409be0ca None None] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:00:44.856377 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cee59b6e-b780-4849-ab29-fc1c409be0ca None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:00:44.856377 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cee59b6e-b780-4849-ab29-fc1c409be0ca None None] PATH_INFO: `/` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:00:44.858308 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 699/1417] 127.0.0.1 () {54 vars in 901 bytes} [Tue Sep 13 16:00:44 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 16:00:44.864613 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8ddd808-03d8-4c51-a55a-a420cbba3377 None None] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:00:44.865095 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8ddd808-03d8-4c51-a55a-a420cbba3377 None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:00:44.865196 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8ddd808-03d8-4c51-a55a-a420cbba3377 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:00:44.946664 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.auth.core [None req-b8ddd808-03d8-4c51-a55a-a420cbba3377 None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81437) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:00:44.947907 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-b8ddd808-03d8-4c51-a55a-a420cbba3377 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:00:44.974609 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:00:45.043943 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 719/1418] 127.0.0.1 () {58 vars in 998 bytes} [Tue Sep 13 16:00:44 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 180 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:00:45.195476 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cee59b6e-b780-4849-ab29-fc1c409be0ca None None] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:00:45.196915 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-cee59b6e-b780-4849-ab29-fc1c409be0ca None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:00:45.343000 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cee59b6e-b780-4849-ab29-fc1c409be0ca None None] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:00:45.383317 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:00:45.456556 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-277bb55c-469a-482b-891c-ce31c40add9d service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4e039b46c5e3445bbb39904b00c5430e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'admin', 'ResellerAdmin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:00:45.457081 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-277bb55c-469a-482b-891c-ce31c40add9d service nova] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:00:45.457387 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-277bb55c-469a-482b-891c-ce31c40add9d service nova] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:00:45.457658 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-277bb55c-469a-482b-891c-ce31c40add9d service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:00:45.459268 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-277bb55c-469a-482b-891c-ce31c40add9d service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:00:45.550457 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-277bb55c-469a-482b-891c-ce31c40add9d service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:00:45.552134 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-277bb55c-469a-482b-891c-ce31c40add9d service nova] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:00:45.569780 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 700/1419] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 16:00:45 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 375 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 16:00:46.711197 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b65f728-ee5f-4cef-87c7-3a4d01695282 None None] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:00:46.711712 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b65f728-ee5f-4cef-87c7-3a4d01695282 None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:00:46.711712 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b65f728-ee5f-4cef-87c7-3a4d01695282 None None] PATH_INFO: `/` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:00:46.712999 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 720/1420] 127.0.0.1 () {54 vars in 901 bytes} [Tue Sep 13 16:00:46 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 16:00:46.719521 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce2f2be1-94b3-463c-9230-bc9cc4046c81 None None] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:00:46.719949 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce2f2be1-94b3-463c-9230-bc9cc4046c81 None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:00:46.720217 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce2f2be1-94b3-463c-9230-bc9cc4046c81 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:00:46.773981 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:00:46.797188 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.auth.core [None req-ce2f2be1-94b3-463c-9230-bc9cc4046c81 None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81438) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:00:46.798396 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-ce2f2be1-94b3-463c-9230-bc9cc4046c81 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:00:46.895145 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 701/1421] 127.0.0.1 () {58 vars in 998 bytes} [Tue Sep 13 16:00:46 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 177 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:00:47.049706 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b65f728-ee5f-4cef-87c7-3a4d01695282 None None] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:00:47.051242 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-9b65f728-ee5f-4cef-87c7-3a4d01695282 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:00:47.195863 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b65f728-ee5f-4cef-87c7-3a4d01695282 None None] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:00:47.208445 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:00:47.308914 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d07f4f6-c61e-4aa5-89ea-b6934f5ea96f service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4e039b46c5e3445bbb39904b00c5430e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'admin', 'ResellerAdmin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:00:47.309431 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d07f4f6-c61e-4aa5-89ea-b6934f5ea96f service nova] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:00:47.309709 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d07f4f6-c61e-4aa5-89ea-b6934f5ea96f service nova] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:00:47.309980 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d07f4f6-c61e-4aa5-89ea-b6934f5ea96f service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:00:47.312232 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-7d07f4f6-c61e-4aa5-89ea-b6934f5ea96f service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:00:47.400817 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d07f4f6-c61e-4aa5-89ea-b6934f5ea96f service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:00:47.402403 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d07f4f6-c61e-4aa5-89ea-b6934f5ea96f service nova] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:00:47.420937 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 721/1422] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 16:00:47 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 372 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 16:00:48.537230 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0086d06c-8c05-4879-9c64-ccee779e5296 None None] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:00:48.537848 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0086d06c-8c05-4879-9c64-ccee779e5296 None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:00:48.537848 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0086d06c-8c05-4879-9c64-ccee779e5296 None None] PATH_INFO: `/` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:00:48.539400 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 702/1423] 127.0.0.1 () {54 vars in 901 bytes} [Tue Sep 13 16:00:48 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 16:00:48.545882 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fa452f7-a136-462c-aed3-0d83d8285e7d None None] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:00:48.546333 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fa452f7-a136-462c-aed3-0d83d8285e7d None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:00:48.546445 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fa452f7-a136-462c-aed3-0d83d8285e7d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:00:48.568301 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:00:48.626184 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.auth.core [None req-8fa452f7-a136-462c-aed3-0d83d8285e7d None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81437) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:00:48.627432 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-8fa452f7-a136-462c-aed3-0d83d8285e7d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:00:48.723897 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 722/1424] 127.0.0.1 () {58 vars in 998 bytes} [Tue Sep 13 16:00:48 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 179 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:00:48.879311 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0086d06c-8c05-4879-9c64-ccee779e5296 None None] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:00:48.880877 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-0086d06c-8c05-4879-9c64-ccee779e5296 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:00:48.989957 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:00:49.025465 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0086d06c-8c05-4879-9c64-ccee779e5296 None None] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:00:49.138305 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c763c5e3-fc24-4026-a7a6-59553e6eb816 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4e039b46c5e3445bbb39904b00c5430e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'admin', 'ResellerAdmin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:00:49.139391 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c763c5e3-fc24-4026-a7a6-59553e6eb816 service nova] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:00:49.139671 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c763c5e3-fc24-4026-a7a6-59553e6eb816 service nova] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:00:49.139941 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c763c5e3-fc24-4026-a7a6-59553e6eb816 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:00:49.141554 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-c763c5e3-fc24-4026-a7a6-59553e6eb816 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:00:49.218263 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:00:49.231855 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c763c5e3-fc24-4026-a7a6-59553e6eb816 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:00:49.233624 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c763c5e3-fc24-4026-a7a6-59553e6eb816 service nova] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:00:49.251416 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 703/1425] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 16:00:48 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 373 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 16:00:50.374060 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0830eac-8dc6-4cbd-994d-bfd196985b4e None None] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:00:50.374631 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0830eac-8dc6-4cbd-994d-bfd196985b4e None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:00:50.374631 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0830eac-8dc6-4cbd-994d-bfd196985b4e None None] PATH_INFO: `/` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:00:50.375965 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 723/1426] 127.0.0.1 () {54 vars in 901 bytes} [Tue Sep 13 16:00:50 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 16:00:50.382106 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30159dfc-a275-49e0-b638-03ef35b91ade None None] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:00:50.382529 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30159dfc-a275-49e0-b638-03ef35b91ade None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:00:50.382692 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30159dfc-a275-49e0-b638-03ef35b91ade None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:00:50.458009 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.auth.core [None req-30159dfc-a275-49e0-b638-03ef35b91ade None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81438) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:00:50.459157 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-30159dfc-a275-49e0-b638-03ef35b91ade None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:00:50.560141 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 704/1427] 127.0.0.1 () {58 vars in 998 bytes} [Tue Sep 13 16:00:50 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 179 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 13 16:00:50.710437 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0830eac-8dc6-4cbd-994d-bfd196985b4e None None] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:00:50.711931 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-e0830eac-8dc6-4cbd-994d-bfd196985b4e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:00:50.795914 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:00:50.857828 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0830eac-8dc6-4cbd-994d-bfd196985b4e None None] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:00:50.974389 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a83a21b7-c389-4271-a8ef-0787d757ca5d service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4e039b46c5e3445bbb39904b00c5430e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'admin', 'ResellerAdmin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:00:50.974938 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a83a21b7-c389-4271-a8ef-0787d757ca5d service nova] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:00:50.975208 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a83a21b7-c389-4271-a8ef-0787d757ca5d service nova] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:00:50.975473 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a83a21b7-c389-4271-a8ef-0787d757ca5d service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:00:50.977042 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-a83a21b7-c389-4271-a8ef-0787d757ca5d service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:00:51.026366 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:00:51.070160 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a83a21b7-c389-4271-a8ef-0787d757ca5d service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:00:51.071786 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a83a21b7-c389-4271-a8ef-0787d757ca5d service nova] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:00:51.089981 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 724/1428] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 16:00:50 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 380 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 16:00:52.148062 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd39df2a-c200-4919-a877-a9e1502b7d1f None None] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:00:52.148641 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd39df2a-c200-4919-a877-a9e1502b7d1f None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:00:52.148641 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd39df2a-c200-4919-a877-a9e1502b7d1f None None] PATH_INFO: `/` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:00:52.149985 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 705/1429] 127.0.0.1 () {54 vars in 901 bytes} [Tue Sep 13 16:00:52 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 16:00:52.156343 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73751a79-1fc9-4d02-ae56-8cf15abfb72a None None] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:00:52.156795 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73751a79-1fc9-4d02-ae56-8cf15abfb72a None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:00:52.156903 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73751a79-1fc9-4d02-ae56-8cf15abfb72a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:00:52.231232 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.auth.core [None req-73751a79-1fc9-4d02-ae56-8cf15abfb72a None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81437) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:00:52.233058 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-73751a79-1fc9-4d02-ae56-8cf15abfb72a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:00:52.327207 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 725/1430] 127.0.0.1 () {58 vars in 998 bytes} [Tue Sep 13 16:00:52 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 172 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:00:52.479653 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd39df2a-c200-4919-a877-a9e1502b7d1f None None] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:00:52.481177 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-fd39df2a-c200-4919-a877-a9e1502b7d1f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:00:52.531533 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:00:52.629346 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd39df2a-c200-4919-a877-a9e1502b7d1f None None] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:00:52.743853 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f038b35e-c27e-40ee-9b54-6b99dba30918 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4e039b46c5e3445bbb39904b00c5430e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'admin', 'ResellerAdmin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:00:52.744462 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f038b35e-c27e-40ee-9b54-6b99dba30918 service nova] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:00:52.744743 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f038b35e-c27e-40ee-9b54-6b99dba30918 service nova] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:00:52.745020 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f038b35e-c27e-40ee-9b54-6b99dba30918 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:00:52.746615 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-f038b35e-c27e-40ee-9b54-6b99dba30918 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:00:52.754345 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:00:52.837725 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f038b35e-c27e-40ee-9b54-6b99dba30918 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:00:52.839336 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f038b35e-c27e-40ee-9b54-6b99dba30918 service nova] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:00:52.857541 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 706/1431] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 16:00:52 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 378 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 16:00:53.906392 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1a8321a-d9ae-4c80-b1a5-4b2daee9b8f1 None None] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:00:53.907005 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1a8321a-d9ae-4c80-b1a5-4b2daee9b8f1 None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:00:53.907005 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1a8321a-d9ae-4c80-b1a5-4b2daee9b8f1 None None] PATH_INFO: `/` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:00:53.908325 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 726/1432] 127.0.0.1 () {54 vars in 901 bytes} [Tue Sep 13 16:00:53 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 16:00:53.914772 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76e88c08-f506-400d-b997-881578097ee0 None None] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:00:53.915250 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76e88c08-f506-400d-b997-881578097ee0 None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:00:53.915596 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76e88c08-f506-400d-b997-881578097ee0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:00:53.990233 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.auth.core [None req-76e88c08-f506-400d-b997-881578097ee0 None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81438) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:00:53.991454 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-76e88c08-f506-400d-b997-881578097ee0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:00:54.059622 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:00:54.090227 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 707/1433] 127.0.0.1 () {58 vars in 998 bytes} [Tue Sep 13 16:00:53 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 176 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:00:54.238681 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1a8321a-d9ae-4c80-b1a5-4b2daee9b8f1 None None] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:00:54.240172 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-b1a8321a-d9ae-4c80-b1a5-4b2daee9b8f1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:00:54.257669 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:00:54.396136 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1a8321a-d9ae-4c80-b1a5-4b2daee9b8f1 None None] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:00:54.484280 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:00:54.511273 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15bce334-0efb-4337-8dfc-94f85006bca4 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4e039b46c5e3445bbb39904b00c5430e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'admin', 'ResellerAdmin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:00:54.511813 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15bce334-0efb-4337-8dfc-94f85006bca4 service nova] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:00:54.512085 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15bce334-0efb-4337-8dfc-94f85006bca4 service nova] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:00:54.512348 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15bce334-0efb-4337-8dfc-94f85006bca4 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:00:54.514322 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-15bce334-0efb-4337-8dfc-94f85006bca4 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:00:54.604850 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15bce334-0efb-4337-8dfc-94f85006bca4 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:00:54.606433 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15bce334-0efb-4337-8dfc-94f85006bca4 service nova] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:00:54.625137 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 727/1434] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 16:00:54 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 387 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 16:00:55.683268 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0f5dc8b-eabc-441a-8d35-20db75fbd433 None None] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:00:55.683843 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0f5dc8b-eabc-441a-8d35-20db75fbd433 None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:00:55.683843 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0f5dc8b-eabc-441a-8d35-20db75fbd433 None None] PATH_INFO: `/` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:00:55.685528 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 708/1435] 127.0.0.1 () {54 vars in 901 bytes} [Tue Sep 13 16:00:55 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 16:00:55.691968 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1b78ce4-d4bc-4060-9d9f-c10a8f36d365 None None] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:00:55.692428 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1b78ce4-d4bc-4060-9d9f-c10a8f36d365 None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:00:55.692530 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1b78ce4-d4bc-4060-9d9f-c10a8f36d365 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:00:55.769304 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.auth.core [None req-e1b78ce4-d4bc-4060-9d9f-c10a8f36d365 None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81437) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:00:55.770552 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-e1b78ce4-d4bc-4060-9d9f-c10a8f36d365 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:00:55.811496 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:00:55.865725 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 728/1436] 127.0.0.1 () {58 vars in 998 bytes} [Tue Sep 13 16:00:55 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 175 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:00:56.016985 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0f5dc8b-eabc-441a-8d35-20db75fbd433 None None] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:00:56.018414 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-f0f5dc8b-eabc-441a-8d35-20db75fbd433 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:00:56.165972 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0f5dc8b-eabc-441a-8d35-20db75fbd433 None None] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:00:56.220485 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:00:56.278072 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7dc898b-1c7d-4ff4-817f-521af76e54ac service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4e039b46c5e3445bbb39904b00c5430e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'admin', 'ResellerAdmin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:00:56.279257 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7dc898b-1c7d-4ff4-817f-521af76e54ac service nova] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:00:56.279533 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7dc898b-1c7d-4ff4-817f-521af76e54ac service nova] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:00:56.279805 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7dc898b-1c7d-4ff4-817f-521af76e54ac service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:00:56.281479 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-b7dc898b-1c7d-4ff4-817f-521af76e54ac service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:00:56.370889 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7dc898b-1c7d-4ff4-817f-521af76e54ac service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:00:56.372706 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7dc898b-1c7d-4ff4-817f-521af76e54ac service nova] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:00:56.390197 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 709/1437] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 16:00:56 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 374 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 16:00:57.446606 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-477d91f2-8b66-4202-89cf-77bf1e00a7c7 None None] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:00:57.447230 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-477d91f2-8b66-4202-89cf-77bf1e00a7c7 None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:00:57.447230 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-477d91f2-8b66-4202-89cf-77bf1e00a7c7 None None] PATH_INFO: `/` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:00:57.448519 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 729/1438] 127.0.0.1 () {54 vars in 901 bytes} [Tue Sep 13 16:00:57 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 16:00:57.454756 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87909766-9e1b-44b1-b949-58131a4a96b9 None None] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:00:57.455245 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87909766-9e1b-44b1-b949-58131a4a96b9 None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:00:57.455351 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87909766-9e1b-44b1-b949-58131a4a96b9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:00:57.530867 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.auth.core [None req-87909766-9e1b-44b1-b949-58131a4a96b9 None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81438) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:00:57.532112 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-87909766-9e1b-44b1-b949-58131a4a96b9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:00:57.534138 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:00:57.630959 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 710/1439] 127.0.0.1 () {58 vars in 998 bytes} [Tue Sep 13 16:00:57 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 177 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:00:57.782205 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-477d91f2-8b66-4202-89cf-77bf1e00a7c7 None None] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:00:57.783980 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-477d91f2-8b66-4202-89cf-77bf1e00a7c7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:00:57.933993 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-477d91f2-8b66-4202-89cf-77bf1e00a7c7 None None] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:00:57.962173 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:00:58.048406 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cd573f3-b129-4d56-b07c-934ffc0503c9 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4e039b46c5e3445bbb39904b00c5430e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'admin', 'ResellerAdmin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:00:58.048939 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cd573f3-b129-4d56-b07c-934ffc0503c9 service nova] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:00:58.049211 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cd573f3-b129-4d56-b07c-934ffc0503c9 service nova] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:00:58.049478 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cd573f3-b129-4d56-b07c-934ffc0503c9 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:00:58.051087 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-4cd573f3-b129-4d56-b07c-934ffc0503c9 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:00:58.141195 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4cd573f3-b129-4d56-b07c-934ffc0503c9 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:00:58.142773 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4cd573f3-b129-4d56-b07c-934ffc0503c9 service nova] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:00:58.160416 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 730/1440] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 16:00:57 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 379 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 16:01:00.295111 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-461ed88d-0cb1-40e6-8889-cc2bc06f76b5 None None] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:01:00.295688 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-461ed88d-0cb1-40e6-8889-cc2bc06f76b5 None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:01:00.295688 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-461ed88d-0cb1-40e6-8889-cc2bc06f76b5 None None] PATH_INFO: `/` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:01:00.296983 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 711/1441] 127.0.0.1 () {54 vars in 917 bytes} [Tue Sep 13 16:01:00 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 16:01:00.302184 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ad5bef9-dd68-4759-b681-b6b45955ec81 None None] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:01:00.302631 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ad5bef9-dd68-4759-b681-b6b45955ec81 None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:01:00.302746 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ad5bef9-dd68-4759-b681-b6b45955ec81 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:01:00.344717 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:01:00.370444 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.auth.core [None req-7ad5bef9-dd68-4759-b681-b6b45955ec81 None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81437) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:01:00.371748 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-7ad5bef9-dd68-4759-b681-b6b45955ec81 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:01:00.468377 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 731/1442] 127.0.0.1 () {58 vars in 1014 bytes} [Tue Sep 13 16:01:00 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 167 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:01:00.765423 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-461ed88d-0cb1-40e6-8889-cc2bc06f76b5 None None] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:01:00.766971 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-461ed88d-0cb1-40e6-8889-cc2bc06f76b5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:01:00.870059 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-461ed88d-0cb1-40e6-8889-cc2bc06f76b5 None None] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:01:00.902310 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:01:00.946974 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b1113bf-6aa6-4ed2-831f-7e90b09ef6cb service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f436f07939d4855a21b195bc7566244', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:01:00.947581 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b1113bf-6aa6-4ed2-831f-7e90b09ef6cb service neutron] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:01:00.947860 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b1113bf-6aa6-4ed2-831f-7e90b09ef6cb service neutron] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:01:00.948131 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b1113bf-6aa6-4ed2-831f-7e90b09ef6cb service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:01:00.950668 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-0b1113bf-6aa6-4ed2-831f-7e90b09ef6cb service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:01:01.060188 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b1113bf-6aa6-4ed2-831f-7e90b09ef6cb service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:01:01.061548 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b1113bf-6aa6-4ed2-831f-7e90b09ef6cb service neutron] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:01:01.080429 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 712/1443] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 16:01:00 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 316 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 16:01:03.442291 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bda43cf-5e70-4d21-85be-c068df770d8b None None] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:01:03.442861 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bda43cf-5e70-4d21-85be-c068df770d8b None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:01:03.442861 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bda43cf-5e70-4d21-85be-c068df770d8b None None] PATH_INFO: `/` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:01:03.444194 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 732/1444] 127.0.0.1 () {54 vars in 917 bytes} [Tue Sep 13 16:01:03 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 16:01:03.449299 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a2a04cc-99de-42f0-9e8c-12773ac6e020 None None] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:01:03.449934 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a2a04cc-99de-42f0-9e8c-12773ac6e020 None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:01:03.449934 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a2a04cc-99de-42f0-9e8c-12773ac6e020 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:01:03.516586 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.auth.core [None req-2a2a04cc-99de-42f0-9e8c-12773ac6e020 None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81438) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:01:03.517798 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-2a2a04cc-99de-42f0-9e8c-12773ac6e020 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:01:03.531021 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:01:03.615654 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 713/1445] 127.0.0.1 () {58 vars in 1014 bytes} [Tue Sep 13 16:01:03 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 167 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:01:03.901658 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bda43cf-5e70-4d21-85be-c068df770d8b None None] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:01:03.903290 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-3bda43cf-5e70-4d21-85be-c068df770d8b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:01:04.004694 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bda43cf-5e70-4d21-85be-c068df770d8b None None] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:01:04.017888 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:01:04.080876 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3f8ea32-e09b-41bc-9c3e-9594ea13a328 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f436f07939d4855a21b195bc7566244', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:01:04.081555 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3f8ea32-e09b-41bc-9c3e-9594ea13a328 service neutron] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:01:04.081834 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3f8ea32-e09b-41bc-9c3e-9594ea13a328 service neutron] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:01:04.082106 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3f8ea32-e09b-41bc-9c3e-9594ea13a328 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:01:04.083725 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-c3f8ea32-e09b-41bc-9c3e-9594ea13a328 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:01:04.190514 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3f8ea32-e09b-41bc-9c3e-9594ea13a328 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:01:04.191902 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3f8ea32-e09b-41bc-9c3e-9594ea13a328 service neutron] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:01:04.210251 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 733/1446] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 16:01:03 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 309 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 16:01:06.681898 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fc15cde-a2ec-4238-9431-6c7c168c8213 None None] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:01:06.682464 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fc15cde-a2ec-4238-9431-6c7c168c8213 None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:01:06.682464 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fc15cde-a2ec-4238-9431-6c7c168c8213 None None] PATH_INFO: `/` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:01:06.683904 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 714/1447] 127.0.0.1 () {54 vars in 917 bytes} [Tue Sep 13 16:01:06 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 16:01:06.689266 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6a9a43e-1c2d-44b0-8532-80a274cd6c22 None None] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:01:06.689689 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6a9a43e-1c2d-44b0-8532-80a274cd6c22 None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:01:06.689795 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6a9a43e-1c2d-44b0-8532-80a274cd6c22 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:01:06.738191 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:01:06.757308 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.auth.core [None req-c6a9a43e-1c2d-44b0-8532-80a274cd6c22 None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81437) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:01:06.758524 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-c6a9a43e-1c2d-44b0-8532-80a274cd6c22 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:01:06.853846 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 734/1448] 127.0.0.1 () {58 vars in 1014 bytes} [Tue Sep 13 16:01:06 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 166 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:01:07.142627 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fc15cde-a2ec-4238-9431-6c7c168c8213 None None] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:01:07.144199 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-2fc15cde-a2ec-4238-9431-6c7c168c8213 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:01:07.246124 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fc15cde-a2ec-4238-9431-6c7c168c8213 None None] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:01:07.285876 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:01:07.319320 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20eec70a-124d-4f2b-bcc0-06cc2ca8fa10 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f436f07939d4855a21b195bc7566244', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:01:07.319922 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20eec70a-124d-4f2b-bcc0-06cc2ca8fa10 service neutron] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:01:07.320200 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20eec70a-124d-4f2b-bcc0-06cc2ca8fa10 service neutron] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:01:07.320470 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20eec70a-124d-4f2b-bcc0-06cc2ca8fa10 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:01:07.322048 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-20eec70a-124d-4f2b-bcc0-06cc2ca8fa10 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:01:07.430736 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20eec70a-124d-4f2b-bcc0-06cc2ca8fa10 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:01:07.432191 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20eec70a-124d-4f2b-bcc0-06cc2ca8fa10 service neutron] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:01:07.450463 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 715/1449] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 16:01:07 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 308 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 16:01:09.735884 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40e0ee87-0e6c-4314-959d-d12f7a80473a None None] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:01:09.736426 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40e0ee87-0e6c-4314-959d-d12f7a80473a None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:01:09.736540 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40e0ee87-0e6c-4314-959d-d12f7a80473a None None] PATH_INFO: `/` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:01:09.737849 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 735/1450] 127.0.0.1 () {54 vars in 917 bytes} [Tue Sep 13 16:01:09 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 16:01:09.743658 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b37985e0-acf8-4a4b-ad6b-87b01b8e9d78 None None] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:01:09.744138 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b37985e0-acf8-4a4b-ad6b-87b01b8e9d78 None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:01:09.744253 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b37985e0-acf8-4a4b-ad6b-87b01b8e9d78 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:01:09.809622 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.auth.core [None req-b37985e0-acf8-4a4b-ad6b-87b01b8e9d78 None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81438) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:01:09.810826 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-b37985e0-acf8-4a4b-ad6b-87b01b8e9d78 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:01:09.830999 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:01:09.908401 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 716/1451] 127.0.0.1 () {58 vars in 1014 bytes} [Tue Sep 13 16:01:09 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 166 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:01:10.197936 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40e0ee87-0e6c-4314-959d-d12f7a80473a None None] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:01:10.199536 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-40e0ee87-0e6c-4314-959d-d12f7a80473a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:01:10.302142 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40e0ee87-0e6c-4314-959d-d12f7a80473a None None] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:01:10.322940 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:01:10.376785 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57d08e02-329f-4e37-8c29-c151fc7220c4 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f436f07939d4855a21b195bc7566244', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:01:10.377393 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57d08e02-329f-4e37-8c29-c151fc7220c4 service neutron] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:01:10.377673 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57d08e02-329f-4e37-8c29-c151fc7220c4 service neutron] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:01:10.377951 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57d08e02-329f-4e37-8c29-c151fc7220c4 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:01:10.379644 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-57d08e02-329f-4e37-8c29-c151fc7220c4 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:01:10.488345 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57d08e02-329f-4e37-8c29-c151fc7220c4 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:01:10.489778 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57d08e02-329f-4e37-8c29-c151fc7220c4 service neutron] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:01:10.507883 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 736/1452] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 16:01:10 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 310 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 16:01:12.712794 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f3749a7-5c9f-4974-b8e9-a409a28ac206 None None] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:01:12.713337 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f3749a7-5c9f-4974-b8e9-a409a28ac206 None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:01:12.713337 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f3749a7-5c9f-4974-b8e9-a409a28ac206 None None] PATH_INFO: `/` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:01:12.715492 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 717/1453] 127.0.0.1 () {54 vars in 917 bytes} [Tue Sep 13 16:01:12 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 16:01:12.720808 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30bbc05a-24f7-4da1-8bf7-491e64d9d36a None None] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:01:12.721192 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30bbc05a-24f7-4da1-8bf7-491e64d9d36a None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:01:12.721357 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30bbc05a-24f7-4da1-8bf7-491e64d9d36a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:01:12.787033 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:01:12.790365 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.auth.core [None req-30bbc05a-24f7-4da1-8bf7-491e64d9d36a None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81437) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:01:12.792088 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-30bbc05a-24f7-4da1-8bf7-491e64d9d36a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:01:12.887287 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 737/1454] 127.0.0.1 () {58 vars in 1014 bytes} [Tue Sep 13 16:01:12 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 167 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:01:13.174841 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f3749a7-5c9f-4974-b8e9-a409a28ac206 None None] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:01:13.176412 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-8f3749a7-5c9f-4974-b8e9-a409a28ac206 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:01:13.280006 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f3749a7-5c9f-4974-b8e9-a409a28ac206 None None] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:01:13.329425 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:01:13.354428 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b675017c-0027-4094-8ac0-b020fa805fca service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f436f07939d4855a21b195bc7566244', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:01:13.355575 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b675017c-0027-4094-8ac0-b020fa805fca service neutron] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:01:13.355855 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b675017c-0027-4094-8ac0-b020fa805fca service neutron] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:01:13.356127 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b675017c-0027-4094-8ac0-b020fa805fca service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:01:13.357719 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-b675017c-0027-4094-8ac0-b020fa805fca service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:01:13.466577 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b675017c-0027-4094-8ac0-b020fa805fca service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:01:13.468043 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b675017c-0027-4094-8ac0-b020fa805fca service neutron] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:01:13.486157 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 718/1455] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 16:01:13 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 312 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 16:01:15.757074 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1c063ba-5523-4a7b-929c-a5fed1a7def2 None None] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:01:15.757652 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1c063ba-5523-4a7b-929c-a5fed1a7def2 None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:01:15.757652 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1c063ba-5523-4a7b-929c-a5fed1a7def2 None None] PATH_INFO: `/` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:01:15.759044 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 738/1456] 127.0.0.1 () {54 vars in 917 bytes} [Tue Sep 13 16:01:15 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 16:01:15.764654 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b331baa9-0fd8-43f0-b37a-6dc34696c327 None None] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:01:15.765051 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b331baa9-0fd8-43f0-b37a-6dc34696c327 None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:01:15.765199 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b331baa9-0fd8-43f0-b37a-6dc34696c327 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:01:15.830713 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.auth.core [None req-b331baa9-0fd8-43f0-b37a-6dc34696c327 None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81438) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:01:15.831988 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-b331baa9-0fd8-43f0-b37a-6dc34696c327 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:01:15.861752 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:01:15.929278 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 719/1457] 127.0.0.1 () {58 vars in 1014 bytes} [Tue Sep 13 16:01:15 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 166 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 13 16:01:16.212729 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1c063ba-5523-4a7b-929c-a5fed1a7def2 None None] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:01:16.214849 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-d1c063ba-5523-4a7b-929c-a5fed1a7def2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:01:16.316578 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1c063ba-5523-4a7b-929c-a5fed1a7def2 None None] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:01:16.345685 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:01:16.393317 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1600d9c-1d17-4b02-8cb7-c0e01b00dd08 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f436f07939d4855a21b195bc7566244', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:01:16.393934 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1600d9c-1d17-4b02-8cb7-c0e01b00dd08 service neutron] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:01:16.394214 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1600d9c-1d17-4b02-8cb7-c0e01b00dd08 service neutron] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:01:16.394491 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1600d9c-1d17-4b02-8cb7-c0e01b00dd08 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:01:16.396124 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-e1600d9c-1d17-4b02-8cb7-c0e01b00dd08 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:01:16.507734 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1600d9c-1d17-4b02-8cb7-c0e01b00dd08 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:01:16.509172 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1600d9c-1d17-4b02-8cb7-c0e01b00dd08 service neutron] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:01:16.527617 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 739/1458] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 16:01:16 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 315 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 16:01:18.770347 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02c1e53e-ff34-4f07-881f-c8e762c2a328 None None] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:01:18.770927 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02c1e53e-ff34-4f07-881f-c8e762c2a328 None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:01:18.770927 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02c1e53e-ff34-4f07-881f-c8e762c2a328 None None] PATH_INFO: `/` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:01:18.772264 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 720/1459] 127.0.0.1 () {54 vars in 917 bytes} [Tue Sep 13 16:01:18 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 16:01:18.777830 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3be4543-23b4-4b84-a19d-d652421d3148 None None] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:01:18.778246 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3be4543-23b4-4b84-a19d-d652421d3148 None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:01:18.778373 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3be4543-23b4-4b84-a19d-d652421d3148 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:01:18.844711 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.auth.core [None req-b3be4543-23b4-4b84-a19d-d652421d3148 None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81437) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:01:18.846016 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-b3be4543-23b4-4b84-a19d-d652421d3148 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:01:18.855505 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:01:18.942629 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 740/1460] 127.0.0.1 () {58 vars in 1014 bytes} [Tue Sep 13 16:01:18 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 166 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 13 16:01:19.223182 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02c1e53e-ff34-4f07-881f-c8e762c2a328 None None] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:01:19.225047 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-02c1e53e-ff34-4f07-881f-c8e762c2a328 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:01:19.329288 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02c1e53e-ff34-4f07-881f-c8e762c2a328 None None] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:01:19.384952 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:01:19.404884 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9cfdb01-d8cc-47f7-bfdc-4061cc4355f4 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f436f07939d4855a21b195bc7566244', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:01:19.405490 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9cfdb01-d8cc-47f7-bfdc-4061cc4355f4 service neutron] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:01:19.405769 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9cfdb01-d8cc-47f7-bfdc-4061cc4355f4 service neutron] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:01:19.406042 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9cfdb01-d8cc-47f7-bfdc-4061cc4355f4 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:01:19.407660 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-f9cfdb01-d8cc-47f7-bfdc-4061cc4355f4 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:01:19.519418 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9cfdb01-d8cc-47f7-bfdc-4061cc4355f4 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:01:19.520755 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9cfdb01-d8cc-47f7-bfdc-4061cc4355f4 service neutron] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:01:19.539357 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 721/1461] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 16:01:19 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 317 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 16:01:21.769597 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-816d1e16-1a02-4892-ab5f-8630edebddb7 None None] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:01:21.770259 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-816d1e16-1a02-4892-ab5f-8630edebddb7 None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:01:21.770259 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-816d1e16-1a02-4892-ab5f-8630edebddb7 None None] PATH_INFO: `/` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:01:21.771507 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 741/1462] 127.0.0.1 () {54 vars in 917 bytes} [Tue Sep 13 16:01:21 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 16:01:21.777331 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e36297f-fb82-4eab-9908-13fead9e436e None None] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:01:21.777790 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e36297f-fb82-4eab-9908-13fead9e436e None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:01:21.777881 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e36297f-fb82-4eab-9908-13fead9e436e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:01:21.849944 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.auth.core [None req-6e36297f-fb82-4eab-9908-13fead9e436e None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81438) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:01:21.851526 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-6e36297f-fb82-4eab-9908-13fead9e436e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:01:21.893142 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:01:21.955690 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 722/1463] 127.0.0.1 () {58 vars in 1014 bytes} [Tue Sep 13 16:01:21 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 180 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:01:22.244169 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-816d1e16-1a02-4892-ab5f-8630edebddb7 None None] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:01:22.245857 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-816d1e16-1a02-4892-ab5f-8630edebddb7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:01:22.348844 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-816d1e16-1a02-4892-ab5f-8630edebddb7 None None] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:01:22.386628 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:01:22.423346 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db3c1bdb-6182-429c-b6a6-d8cb4ab40112 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f436f07939d4855a21b195bc7566244', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:01:22.423949 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db3c1bdb-6182-429c-b6a6-d8cb4ab40112 service neutron] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:01:22.424229 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db3c1bdb-6182-429c-b6a6-d8cb4ab40112 service neutron] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:01:22.424502 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db3c1bdb-6182-429c-b6a6-d8cb4ab40112 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:01:22.426092 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-db3c1bdb-6182-429c-b6a6-d8cb4ab40112 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:01:22.534400 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db3c1bdb-6182-429c-b6a6-d8cb4ab40112 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:01:22.536113 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db3c1bdb-6182-429c-b6a6-d8cb4ab40112 service neutron] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:01:22.553708 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 742/1464] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 16:01:22 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 310 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 16:01:24.823296 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3841a6d-0990-4f14-a3a6-6c2ce1742957 None None] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:01:24.823868 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3841a6d-0990-4f14-a3a6-6c2ce1742957 None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:01:24.823868 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3841a6d-0990-4f14-a3a6-6c2ce1742957 None None] PATH_INFO: `/` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:01:24.825185 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 723/1465] 127.0.0.1 () {54 vars in 917 bytes} [Tue Sep 13 16:01:24 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 16:01:24.831057 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e223649-3c01-483b-b04d-02f84c890ece None None] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:01:24.831504 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e223649-3c01-483b-b04d-02f84c890ece None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:01:24.831613 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e223649-3c01-483b-b04d-02f84c890ece None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:01:24.897667 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.auth.core [None req-3e223649-3c01-483b-b04d-02f84c890ece None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81437) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:01:24.898879 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-3e223649-3c01-483b-b04d-02f84c890ece None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:01:24.917111 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:01:25.001577 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 743/1466] 127.0.0.1 () {58 vars in 1014 bytes} [Tue Sep 13 16:01:24 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 171 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:01:25.286174 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3841a6d-0990-4f14-a3a6-6c2ce1742957 None None] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:01:25.287740 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-b3841a6d-0990-4f14-a3a6-6c2ce1742957 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:01:25.391489 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3841a6d-0990-4f14-a3a6-6c2ce1742957 None None] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:01:25.458352 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:01:25.465382 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7ae2f59-d33d-49b8-980d-1df6fab9dcc6 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f436f07939d4855a21b195bc7566244', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:01:25.465983 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7ae2f59-d33d-49b8-980d-1df6fab9dcc6 service neutron] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:01:25.466260 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7ae2f59-d33d-49b8-980d-1df6fab9dcc6 service neutron] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:01:25.466534 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7ae2f59-d33d-49b8-980d-1df6fab9dcc6 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:01:25.468145 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-e7ae2f59-d33d-49b8-980d-1df6fab9dcc6 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:01:25.577189 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7ae2f59-d33d-49b8-980d-1df6fab9dcc6 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:01:25.578611 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7ae2f59-d33d-49b8-980d-1df6fab9dcc6 service neutron] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:01:25.718149 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 724/1467] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 16:01:25 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 433 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 16:01:27.950981 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57460aaa-9d42-42b0-bb7e-2a51eb68e13d None None] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:01:27.951517 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57460aaa-9d42-42b0-bb7e-2a51eb68e13d None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:01:27.951517 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57460aaa-9d42-42b0-bb7e-2a51eb68e13d None None] PATH_INFO: `/` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:01:27.952883 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 744/1468] 127.0.0.1 () {54 vars in 917 bytes} [Tue Sep 13 16:01:27 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 16:01:27.958144 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71b70664-31c0-4ac0-ae96-cc2f8afed6cd None None] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:01:27.958589 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71b70664-31c0-4ac0-ae96-cc2f8afed6cd None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:01:27.958700 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71b70664-31c0-4ac0-ae96-cc2f8afed6cd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:01:28.025148 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.auth.core [None req-71b70664-31c0-4ac0-ae96-cc2f8afed6cd None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81438) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:01:28.026352 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-71b70664-31c0-4ac0-ae96-cc2f8afed6cd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:01:28.075646 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:01:28.124355 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 725/1469] 127.0.0.1 () {58 vars in 1014 bytes} [Tue Sep 13 16:01:27 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 167 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:01:28.407993 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57460aaa-9d42-42b0-bb7e-2a51eb68e13d None None] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:01:28.409510 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-57460aaa-9d42-42b0-bb7e-2a51eb68e13d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:01:28.511933 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57460aaa-9d42-42b0-bb7e-2a51eb68e13d None None] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:01:28.559894 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:01:28.585111 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b9370f8-930c-4462-8f03-6b182026c4bf service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f436f07939d4855a21b195bc7566244', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:01:28.585739 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b9370f8-930c-4462-8f03-6b182026c4bf service neutron] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:01:28.586019 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b9370f8-930c-4462-8f03-6b182026c4bf service neutron] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:01:28.586291 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b9370f8-930c-4462-8f03-6b182026c4bf service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:01:28.588532 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-6b9370f8-930c-4462-8f03-6b182026c4bf service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:01:28.696144 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b9370f8-930c-4462-8f03-6b182026c4bf service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:01:28.697669 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b9370f8-930c-4462-8f03-6b182026c4bf service neutron] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:01:28.716273 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 745/1470] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 16:01:28 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 309 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 16:01:38.122849 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=22 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:01:38.714011 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=66 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:01:48.680263 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9afb3d25-112e-4752-bc3d-fe15cd944e45 None None] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:01:48.680824 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9afb3d25-112e-4752-bc3d-fe15cd944e45 None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:01:48.680824 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9afb3d25-112e-4752-bc3d-fe15cd944e45 None None] PATH_INFO: `/` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:01:48.682157 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 726/1471] 10.222.0.50 () {54 vars in 927 bytes} [Tue Sep 13 16:01:48 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 16:01:48.687433 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-800334ce-d805-43fb-88b4-cf9cc1d8da56 None None] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:01:48.687879 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-800334ce-d805-43fb-88b4-cf9cc1d8da56 None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:01:48.687981 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-800334ce-d805-43fb-88b4-cf9cc1d8da56 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:01:48.753474 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.auth.core [None req-800334ce-d805-43fb-88b4-cf9cc1d8da56 None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81437) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:01:48.754669 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-800334ce-d805-43fb-88b4-cf9cc1d8da56 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:01:48.853932 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 746/1472] 10.222.0.50 () {58 vars in 1024 bytes} [Tue Sep 13 16:01:48 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 167 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:01:49.159804 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26e80803-3475-4ecc-8eb1-5f368f59416e None None] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:01:49.160136 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26e80803-3475-4ecc-8eb1-5f368f59416e None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:01:49.160409 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26e80803-3475-4ecc-8eb1-5f368f59416e None None] PATH_INFO: `/` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:01:49.161744 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 727/1473] 10.222.0.50 () {54 vars in 966 bytes} [Tue Sep 13 16:01:49 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 16:01:49.166782 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4f91ba9-c0aa-4dfc-aa88-2029a2003ddb None None] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:01:49.167094 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4f91ba9-c0aa-4dfc-aa88-2029a2003ddb None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:01:49.167373 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4f91ba9-c0aa-4dfc-aa88-2029a2003ddb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:01:49.243758 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.auth.core [None req-e4f91ba9-c0aa-4dfc-aa88-2029a2003ddb None None] MFA Rules not processed for user `6ab5f8018f464be69d4af7683f553738`. Rule list: `[]` (Enabled: `True`). {{(pid=81437) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:01:49.244963 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-e4f91ba9-c0aa-4dfc-aa88-2029a2003ddb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:01:49.265988 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:01:49.321534 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 747/1474] 10.222.0.50 () {58 vars in 1063 bytes} [Tue Sep 13 16:01:49 2022] POST /identity/v3/auth/tokens => generated 3120 bytes in 156 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:01:49.329376 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26e80803-3475-4ecc-8eb1-5f368f59416e None None] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:01:49.330868 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-26e80803-3475-4ecc-8eb1-5f368f59416e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:01:49.434064 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26e80803-3475-4ecc-8eb1-5f368f59416e None None] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:01:49.505287 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4df43885-6b5c-452c-9f37-b8e796a08145 service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6ab5f8018f464be69d4af7683f553738', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:01:49.505887 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4df43885-6b5c-452c-9f37-b8e796a08145 service glance] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:01:49.506166 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4df43885-6b5c-452c-9f37-b8e796a08145 service glance] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:01:49.506435 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4df43885-6b5c-452c-9f37-b8e796a08145 service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:01:49.508151 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-4df43885-6b5c-452c-9f37-b8e796a08145 service glance] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:01:49.563251 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:01:49.618130 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4df43885-6b5c-452c-9f37-b8e796a08145 service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:01:49.619608 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4df43885-6b5c-452c-9f37-b8e796a08145 service glance] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:01:49.637696 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 728/1475] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 16:01:49 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 309 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 16:01:51.901850 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb710c72-0ab5-43b2-b0d4-68d589a7cb41 None None] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:01:51.902489 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb710c72-0ab5-43b2-b0d4-68d589a7cb41 None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:01:51.902489 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb710c72-0ab5-43b2-b0d4-68d589a7cb41 None None] PATH_INFO: `/` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:01:51.903783 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 748/1476] 10.222.0.50 () {54 vars in 927 bytes} [Tue Sep 13 16:01:51 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 16:01:51.909329 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18d61ba5-5a7d-492b-9237-710c7b38d040 None None] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:01:51.909762 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18d61ba5-5a7d-492b-9237-710c7b38d040 None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:01:51.909884 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18d61ba5-5a7d-492b-9237-710c7b38d040 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:01:51.974697 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.auth.core [None req-18d61ba5-5a7d-492b-9237-710c7b38d040 None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81438) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:01:51.976573 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-18d61ba5-5a7d-492b-9237-710c7b38d040 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:01:52.072040 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 729/1477] 10.222.0.50 () {58 vars in 1024 bytes} [Tue Sep 13 16:01:51 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 164 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 13 16:01:52.375044 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb710c72-0ab5-43b2-b0d4-68d589a7cb41 None None] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:01:52.376574 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-fb710c72-0ab5-43b2-b0d4-68d589a7cb41 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:01:52.503146 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb710c72-0ab5-43b2-b0d4-68d589a7cb41 None None] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:01:52.557597 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:01:52.601367 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb83370f-c570-46a1-9c41-cc66a674ce5d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:01:52.601974 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb83370f-c570-46a1-9c41-cc66a674ce5d admin admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:01:52.602250 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb83370f-c570-46a1-9c41-cc66a674ce5d admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:01:52.602523 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb83370f-c570-46a1-9c41-cc66a674ce5d admin admin] PATH_INFO: `/v3/projects` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:01:52.603194 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb83370f-c570-46a1-9c41-cc66a674ce5d admin admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:01:52.604921 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: WARNING py.warnings [None req-bb83370f-c570-46a1-9c41-cc66a674ce5d admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_projects": "(role:reader and system_scope:all) or (role:reader and domain_id:%(target.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Sep 13 16:01:52.604921 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: warnings.warn(msg) Sep 13 16:01:52.604921 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: Sep 13 16:01:52.606089 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb83370f-c570-46a1-9c41-cc66a674ce5d admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:01:52.624709 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 749/1478] 10.222.0.50 () {56 vars in 1108 bytes} [Tue Sep 13 16:01:52 2022] GET /identity/v3/projects => generated 2553 bytes in 250 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 16:01:54.776179 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b8f1004-a27d-44f3-bffc-db73bb8cfccc None None] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:01:54.776787 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b8f1004-a27d-44f3-bffc-db73bb8cfccc None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:01:54.776787 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b8f1004-a27d-44f3-bffc-db73bb8cfccc None None] PATH_INFO: `/` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:01:54.778052 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 730/1479] 10.222.0.50 () {54 vars in 927 bytes} [Tue Sep 13 16:01:54 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 16:01:54.783263 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a127d60-2a92-4393-bc09-23a7dbec7568 None None] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:01:54.783966 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a127d60-2a92-4393-bc09-23a7dbec7568 None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:01:54.784072 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a127d60-2a92-4393-bc09-23a7dbec7568 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:01:54.853766 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.auth.core [None req-3a127d60-2a92-4393-bc09-23a7dbec7568 None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81437) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:01:54.855018 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-3a127d60-2a92-4393-bc09-23a7dbec7568 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:01:54.951120 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 750/1480] 10.222.0.50 () {58 vars in 1024 bytes} [Tue Sep 13 16:01:54 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 169 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:01:55.242086 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b8f1004-a27d-44f3-bffc-db73bb8cfccc None None] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:01:55.243863 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-7b8f1004-a27d-44f3-bffc-db73bb8cfccc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:01:55.253745 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:01:55.389490 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b8f1004-a27d-44f3-bffc-db73bb8cfccc None None] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:01:55.468611 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:01:55.500254 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38a3e816-d8c8-4487-9162-d8dbefaf79cc service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4e039b46c5e3445bbb39904b00c5430e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'admin', 'ResellerAdmin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:01:55.500854 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38a3e816-d8c8-4487-9162-d8dbefaf79cc service nova] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:01:55.501128 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38a3e816-d8c8-4487-9162-d8dbefaf79cc service nova] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:01:55.501395 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38a3e816-d8c8-4487-9162-d8dbefaf79cc service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:01:55.502981 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-38a3e816-d8c8-4487-9162-d8dbefaf79cc service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:01:55.591657 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38a3e816-d8c8-4487-9162-d8dbefaf79cc service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:01:55.593229 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38a3e816-d8c8-4487-9162-d8dbefaf79cc service nova] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:01:55.610611 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 731/1481] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 16:01:55 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 369 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 16:01:57.953488 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e0718c8-4320-447b-b360-3641441c466a None None] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:01:57.954052 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e0718c8-4320-447b-b360-3641441c466a None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:01:57.954052 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e0718c8-4320-447b-b360-3641441c466a None None] PATH_INFO: `/` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:01:57.955546 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 751/1482] 10.222.0.50 () {54 vars in 927 bytes} [Tue Sep 13 16:01:57 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 16:01:57.961322 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad9872a8-bd7e-4c27-889b-ecd56589ee58 None None] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:01:57.961773 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad9872a8-bd7e-4c27-889b-ecd56589ee58 None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:01:57.961948 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad9872a8-bd7e-4c27-889b-ecd56589ee58 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:01:58.028768 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.auth.core [None req-ad9872a8-bd7e-4c27-889b-ecd56589ee58 None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81438) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:01:58.029985 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-ad9872a8-bd7e-4c27-889b-ecd56589ee58 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:01:58.077837 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:01:58.132046 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 732/1483] 10.222.0.50 () {58 vars in 1024 bytes} [Tue Sep 13 16:01:57 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 172 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:01:58.420427 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e0718c8-4320-447b-b360-3641441c466a None None] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:01:58.421949 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-5e0718c8-4320-447b-b360-3641441c466a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:01:58.445755 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:01:58.525997 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e0718c8-4320-447b-b360-3641441c466a None None] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:01:58.600557 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85837519-4e4d-42c9-b155-b35de45e494c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f436f07939d4855a21b195bc7566244', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:01:58.601148 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85837519-4e4d-42c9-b155-b35de45e494c service neutron] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:01:58.601424 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85837519-4e4d-42c9-b155-b35de45e494c service neutron] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:01:58.601693 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85837519-4e4d-42c9-b155-b35de45e494c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:01:58.603296 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-85837519-4e4d-42c9-b155-b35de45e494c service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:01:58.679528 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:01:58.715030 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85837519-4e4d-42c9-b155-b35de45e494c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:01:58.716494 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85837519-4e4d-42c9-b155-b35de45e494c service neutron] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:01:58.733997 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 752/1484] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 16:01:58 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 314 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 16:02:00.962210 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bd0bc61-f512-4a64-80e0-b0390a66d0aa None None] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:02:00.962754 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bd0bc61-f512-4a64-80e0-b0390a66d0aa None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:02:00.962754 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bd0bc61-f512-4a64-80e0-b0390a66d0aa None None] PATH_INFO: `/` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:02:00.964118 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 733/1485] 10.222.0.50 () {54 vars in 927 bytes} [Tue Sep 13 16:02:00 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 16:02:00.969523 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f360ad2d-d1b8-4385-8b1c-e05f23c317d1 None None] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:02:00.969949 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f360ad2d-d1b8-4385-8b1c-e05f23c317d1 None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:02:00.970080 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f360ad2d-d1b8-4385-8b1c-e05f23c317d1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:02:01.036377 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.auth.core [None req-f360ad2d-d1b8-4385-8b1c-e05f23c317d1 None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81437) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:02:01.038373 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-f360ad2d-d1b8-4385-8b1c-e05f23c317d1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:02:01.133984 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 753/1486] 10.222.0.50 () {58 vars in 1024 bytes} [Tue Sep 13 16:02:00 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 165 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:02:01.425585 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bd0bc61-f512-4a64-80e0-b0390a66d0aa None None] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:02:01.427173 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-1bd0bc61-f512-4a64-80e0-b0390a66d0aa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:02:01.529145 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bd0bc61-f512-4a64-80e0-b0390a66d0aa None None] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:02:01.600356 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:02:01.601823 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cd63429-3a81-4117-a1bb-cf7a81354ef3 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f436f07939d4855a21b195bc7566244', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:02:01.602435 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cd63429-3a81-4117-a1bb-cf7a81354ef3 service neutron] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:02:01.602928 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cd63429-3a81-4117-a1bb-cf7a81354ef3 service neutron] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:02:01.603209 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cd63429-3a81-4117-a1bb-cf7a81354ef3 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:02:01.605147 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-1cd63429-3a81-4117-a1bb-cf7a81354ef3 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:02:01.714043 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1cd63429-3a81-4117-a1bb-cf7a81354ef3 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:02:01.715510 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1cd63429-3a81-4117-a1bb-cf7a81354ef3 service neutron] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:02:01.732888 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 734/1487] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 16:02:01 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 308 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 16:02:04.043451 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8903705c-7286-413a-b74f-382a8818c90f None None] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:02:04.044040 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8903705c-7286-413a-b74f-382a8818c90f None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:02:04.044040 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8903705c-7286-413a-b74f-382a8818c90f None None] PATH_INFO: `/` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:02:04.045315 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 754/1488] 10.222.0.50 () {54 vars in 927 bytes} [Tue Sep 13 16:02:04 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 16:02:04.050554 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb69bc6e-c082-4f3d-94db-62b5fa6b8301 None None] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:02:04.051014 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb69bc6e-c082-4f3d-94db-62b5fa6b8301 None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:02:04.051128 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb69bc6e-c082-4f3d-94db-62b5fa6b8301 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:02:04.118109 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.auth.core [None req-eb69bc6e-c082-4f3d-94db-62b5fa6b8301 None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81438) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:02:04.119356 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-eb69bc6e-c082-4f3d-94db-62b5fa6b8301 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:02:04.171260 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:02:04.217599 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 735/1489] 10.222.0.50 () {58 vars in 1024 bytes} [Tue Sep 13 16:02:04 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 168 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:02:04.524483 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8903705c-7286-413a-b74f-382a8818c90f None None] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:02:04.526047 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-8903705c-7286-413a-b74f-382a8818c90f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:02:04.561642 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:02:04.628511 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8903705c-7286-413a-b74f-382a8818c90f None None] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:02:04.700691 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b579ac6e-ca37-4034-bbdd-e75069184f7c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f436f07939d4855a21b195bc7566244', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:02:04.701298 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b579ac6e-ca37-4034-bbdd-e75069184f7c service neutron] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:02:04.701577 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b579ac6e-ca37-4034-bbdd-e75069184f7c service neutron] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:02:04.701843 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b579ac6e-ca37-4034-bbdd-e75069184f7c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:02:04.703445 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-b579ac6e-ca37-4034-bbdd-e75069184f7c service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:02:04.784250 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:02:04.813625 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b579ac6e-ca37-4034-bbdd-e75069184f7c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:02:04.815096 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b579ac6e-ca37-4034-bbdd-e75069184f7c service neutron] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:02:04.833171 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 755/1490] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 16:02:04 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 309 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 16:02:07.178558 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-678bdf6e-fdd7-4d36-987b-36c0f15a3fb5 None None] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:02:07.179156 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-678bdf6e-fdd7-4d36-987b-36c0f15a3fb5 None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:02:07.179156 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-678bdf6e-fdd7-4d36-987b-36c0f15a3fb5 None None] PATH_INFO: `/` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:02:07.180477 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 736/1491] 10.222.0.50 () {54 vars in 927 bytes} [Tue Sep 13 16:02:07 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 16:02:07.185852 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-637b2608-838d-4962-8009-bc8a294e78c3 None None] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:02:07.186285 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-637b2608-838d-4962-8009-bc8a294e78c3 None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:02:07.186405 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-637b2608-838d-4962-8009-bc8a294e78c3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:02:07.251998 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.auth.core [None req-637b2608-838d-4962-8009-bc8a294e78c3 None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81437) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:02:07.253216 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-637b2608-838d-4962-8009-bc8a294e78c3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:02:07.348302 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 756/1492] 10.222.0.50 () {58 vars in 1024 bytes} [Tue Sep 13 16:02:07 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 163 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:02:07.637127 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-678bdf6e-fdd7-4d36-987b-36c0f15a3fb5 None None] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:02:07.638652 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-678bdf6e-fdd7-4d36-987b-36c0f15a3fb5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:02:07.741904 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-678bdf6e-fdd7-4d36-987b-36c0f15a3fb5 None None] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:02:07.814652 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-147b14fd-f3aa-47b1-a96c-44579d70bb5d service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f436f07939d4855a21b195bc7566244', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:02:07.815277 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-147b14fd-f3aa-47b1-a96c-44579d70bb5d service neutron] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:02:07.815553 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-147b14fd-f3aa-47b1-a96c-44579d70bb5d service neutron] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:02:07.815821 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-147b14fd-f3aa-47b1-a96c-44579d70bb5d service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:02:07.817384 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-147b14fd-f3aa-47b1-a96c-44579d70bb5d service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:02:07.825770 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:02:07.927732 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-147b14fd-f3aa-47b1-a96c-44579d70bb5d service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:02:07.929141 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-147b14fd-f3aa-47b1-a96c-44579d70bb5d service neutron] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:02:07.947859 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 737/1493] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 16:02:07 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 311 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 16:02:17.346770 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=82 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:02:17.945565 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=50 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:03:29.644531 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-893461fc-db4f-4b2d-aedd-586ba0f3e155 None None] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:03:29.645151 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-893461fc-db4f-4b2d-aedd-586ba0f3e155 None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:03:29.645151 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-893461fc-db4f-4b2d-aedd-586ba0f3e155 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:03:29.720495 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.auth.core [None req-893461fc-db4f-4b2d-aedd-586ba0f3e155 None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81437) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:03:29.721701 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-893461fc-db4f-4b2d-aedd-586ba0f3e155 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:03:29.817933 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 757/1494] 10.222.0.50 () {58 vars in 958 bytes} [Tue Sep 13 16:03:29 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 174 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:03:29.826277 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62d1439a-e346-49a9-8a4e-a5ddfdabf7a6 None None] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:03:29.826792 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62d1439a-e346-49a9-8a4e-a5ddfdabf7a6 None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:03:29.826927 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62d1439a-e346-49a9-8a4e-a5ddfdabf7a6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:03:29.901329 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.auth.core [None req-62d1439a-e346-49a9-8a4e-a5ddfdabf7a6 None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81438) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:03:29.902713 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-62d1439a-e346-49a9-8a4e-a5ddfdabf7a6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:03:29.999435 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 738/1495] 10.222.0.50 () {58 vars in 958 bytes} [Tue Sep 13 16:03:29 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 174 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:03:30.004132 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-893461fc-db4f-4b2d-aedd-586ba0f3e155 None None] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:03:30.005333 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-893461fc-db4f-4b2d-aedd-586ba0f3e155 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:03:30.017358 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-893461fc-db4f-4b2d-aedd-586ba0f3e155 None None] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:03:30.080014 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:03:30.113727 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fba590d4-3d3a-4d07-b20a-e0b9a700288f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:03:30.114338 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fba590d4-3d3a-4d07-b20a-e0b9a700288f admin admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:03:30.114652 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fba590d4-3d3a-4d07-b20a-e0b9a700288f admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:03:30.114942 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fba590d4-3d3a-4d07-b20a-e0b9a700288f admin admin] PATH_INFO: `/v3/domains` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:03:30.115607 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fba590d4-3d3a-4d07-b20a-e0b9a700288f admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81437) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 16:03:30.115929 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fba590d4-3d3a-4d07-b20a-e0b9a700288f admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:03:30.116939 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: WARNING py.warnings [None req-fba590d4-3d3a-4d07-b20a-e0b9a700288f admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_domains": "role:reader and system_scope:all" failed scope check. The token used to make the request was project scoped but the policy requires ['system'] scope. This behavior may change in the future where using the intended scope is required Sep 13 16:03:30.116939 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: warnings.warn(msg) Sep 13 16:03:30.116939 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: Sep 13 16:03:30.117829 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fba590d4-3d3a-4d07-b20a-e0b9a700288f admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:03:30.130543 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 758/1496] 10.222.0.50 () {58 vars in 1153 bytes} [Tue Sep 13 16:03:30 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 127 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 16:03:30.139380 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62d1439a-e346-49a9-8a4e-a5ddfdabf7a6 None None] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:03:30.148470 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62d1439a-e346-49a9-8a4e-a5ddfdabf7a6 None None] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:03:30.214202 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:03:30.248750 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f5b2717-eac2-468a-8c02-497368bc75dc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:03:30.249350 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f5b2717-eac2-468a-8c02-497368bc75dc admin admin] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:03:30.249629 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f5b2717-eac2-468a-8c02-497368bc75dc admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:03:30.249901 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f5b2717-eac2-468a-8c02-497368bc75dc admin admin] PATH_INFO: `/v3/projects` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:03:30.250533 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f5b2717-eac2-468a-8c02-497368bc75dc admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:03:30.252307 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: WARNING py.warnings [None req-5f5b2717-eac2-468a-8c02-497368bc75dc admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_project": "(role:admin and system_scope:all) or (role:admin and domain_id:%(target.project.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Sep 13 16:03:30.252307 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: warnings.warn(msg) Sep 13 16:03:30.252307 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: Sep 13 16:03:30.253490 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f5b2717-eac2-468a-8c02-497368bc75dc admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:03:30.309097 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 739/1497] 10.222.0.50 () {60 vars in 1153 bytes} [Tue Sep 13 16:03:30 2022] POST /identity/v3/projects => generated 377 bytes in 171 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 16:03:30.313940 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fba590d4-3d3a-4d07-b20a-e0b9a700288f admin admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:03:30.315545 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-fba590d4-3d3a-4d07-b20a-e0b9a700288f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:03:30.348999 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fba590d4-3d3a-4d07-b20a-e0b9a700288f admin admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:03:30.442438 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e0658d6-dd5d-44ad-893f-b67a2734d6c6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:03:30.442991 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e0658d6-dd5d-44ad-893f-b67a2734d6c6 admin admin] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:03:30.443268 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e0658d6-dd5d-44ad-893f-b67a2734d6c6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:03:30.443542 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e0658d6-dd5d-44ad-893f-b67a2734d6c6 admin admin] PATH_INFO: `/v3/users` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:03:30.444181 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e0658d6-dd5d-44ad-893f-b67a2734d6c6 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:03:30.445194 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: WARNING py.warnings [None req-6e0658d6-dd5d-44ad-893f-b67a2734d6c6 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_user": "(role:admin and system_scope:all) or (role:admin and token.domain.id:%(target.user.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Sep 13 16:03:30.445194 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: warnings.warn(msg) Sep 13 16:03:30.445194 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: Sep 13 16:03:30.446421 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e0658d6-dd5d-44ad-893f-b67a2734d6c6 admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:03:30.489919 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 759/1498] 10.222.0.50 () {60 vars in 1144 bytes} [Tue Sep 13 16:03:30 2022] POST /identity/v3/users => generated 289 bytes in 176 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 16:03:30.494029 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f5b2717-eac2-468a-8c02-497368bc75dc admin admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:03:30.495611 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-5f5b2717-eac2-468a-8c02-497368bc75dc admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:03:30.557248 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f5b2717-eac2-468a-8c02-497368bc75dc admin admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:03:30.640891 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:03:30.653064 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-718d98f3-e065-4fb3-abea-d2af5d9eec56 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:03:30.654288 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-718d98f3-e065-4fb3-abea-d2af5d9eec56 admin admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:03:30.654562 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-718d98f3-e065-4fb3-abea-d2af5d9eec56 admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:03:30.654853 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-718d98f3-e065-4fb3-abea-d2af5d9eec56 admin admin] PATH_INFO: `/v3/roles` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:03:30.655522 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-718d98f3-e065-4fb3-abea-d2af5d9eec56 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:03:30.656531 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: WARNING py.warnings [None req-718d98f3-e065-4fb3-abea-d2af5d9eec56 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_roles": "role:reader and system_scope:all" failed scope check. The token used to make the request was project scoped but the policy requires ['system'] scope. This behavior may change in the future where using the intended scope is required Sep 13 16:03:30.656531 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: warnings.warn(msg) Sep 13 16:03:30.656531 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: Sep 13 16:03:30.657416 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-718d98f3-e065-4fb3-abea-d2af5d9eec56 admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:03:30.670427 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 740/1499] 10.222.0.50 () {58 vars in 1122 bytes} [Tue Sep 13 16:03:30 2022] GET /identity/v3/roles => generated 1420 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 16:03:30.674627 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e0658d6-dd5d-44ad-893f-b67a2734d6c6 admin admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:03:30.676258 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-6e0658d6-dd5d-44ad-893f-b67a2734d6c6 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:03:30.687769 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e0658d6-dd5d-44ad-893f-b67a2734d6c6 admin admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:03:30.713109 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:03:30.783654 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e6e263c-1c73-445b-8d37-b21b90f68f09 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:03:30.784257 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e6e263c-1c73-445b-8d37-b21b90f68f09 admin admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:03:30.784541 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e6e263c-1c73-445b-8d37-b21b90f68f09 admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:03:30.784843 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e6e263c-1c73-445b-8d37-b21b90f68f09 admin admin] PATH_INFO: `/v3/roles` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:03:30.785476 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e6e263c-1c73-445b-8d37-b21b90f68f09 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:03:30.786485 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: WARNING py.warnings [None req-8e6e263c-1c73-445b-8d37-b21b90f68f09 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_roles": "role:reader and system_scope:all" failed scope check. The token used to make the request was project scoped but the policy requires ['system'] scope. This behavior may change in the future where using the intended scope is required Sep 13 16:03:30.786485 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: warnings.warn(msg) Sep 13 16:03:30.786485 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: Sep 13 16:03:30.787396 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e6e263c-1c73-445b-8d37-b21b90f68f09 admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:03:30.799775 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 760/1500] 10.222.0.50 () {58 vars in 1122 bytes} [Tue Sep 13 16:03:30 2022] GET /identity/v3/roles => generated 1420 bytes in 126 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 16:03:30.803957 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-718d98f3-e065-4fb3-abea-d2af5d9eec56 admin admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:03:30.805509 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-718d98f3-e065-4fb3-abea-d2af5d9eec56 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:03:30.838582 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-718d98f3-e065-4fb3-abea-d2af5d9eec56 admin admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:03:30.932804 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3a3b4b1-192f-4540-94bb-74a377259771 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:03:30.933457 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3a3b4b1-192f-4540-94bb-74a377259771 admin admin] REQUEST_METHOD: `PUT` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:03:30.933731 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3a3b4b1-192f-4540-94bb-74a377259771 admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:03:30.933999 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3a3b4b1-192f-4540-94bb-74a377259771 admin admin] PATH_INFO: `/v3/projects/f20a7fb776934767baec85df294a0cc4/users/337fe2ee2e244b7fa126b7280d1e4d60/roles/15db4aaca3264ae29bde38903ba43471` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:03:30.956484 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3a3b4b1-192f-4540-94bb-74a377259771 admin admin] RBAC: Authorizing `identity:create_grant(project_id=f20a7fb776934767baec85df294a0cc4, user_id=337fe2ee2e244b7fa126b7280d1e4d60, role_id=15db4aaca3264ae29bde38903ba43471)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:03:30.957510 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: WARNING py.warnings [None req-c3a3b4b1-192f-4540-94bb-74a377259771 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_grant": "(role:admin and system_scope:all) or ((role:admin and domain_id:%(target.user.domain_id)s and domain_id:%(target.project.domain_id)s) or (role:admin and domain_id:%(target.user.domain_id)s and domain_id:%(target.domain.id)s) or (role:admin and domain_id:%(target.group.domain_id)s and domain_id:%(target.project.domain_id)s) or (role:admin and domain_id:%(target.group.domain_id)s and domain_id:%(target.domain.id)s)) and (domain_id:%(target.role.domain_id)s or None:%(target.role.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Sep 13 16:03:30.957510 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: warnings.warn(msg) Sep 13 16:03:30.957510 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: Sep 13 16:03:30.959463 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3a3b4b1-192f-4540-94bb-74a377259771 admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:03:30.983202 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 741/1501] 10.222.0.50 () {60 vars in 1483 bytes} [Tue Sep 13 16:03:30 2022] PUT /identity/v3/projects/f20a7fb776934767baec85df294a0cc4/users/337fe2ee2e244b7fa126b7280d1e4d60/roles/15db4aaca3264ae29bde38903ba43471 => generated 0 bytes in 180 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 13 16:03:30.995490 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05a2ec6d-0ad2-4360-b09e-f55bd766e60e None None] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:03:30.995777 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05a2ec6d-0ad2-4360-b09e-f55bd766e60e None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:03:30.996053 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05a2ec6d-0ad2-4360-b09e-f55bd766e60e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:03:31.054042 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.auth.core [None req-05a2ec6d-0ad2-4360-b09e-f55bd766e60e None None] MFA Rules not processed for user `337fe2ee2e244b7fa126b7280d1e4d60`. Rule list: `[]` (Enabled: `True`). {{(pid=81437) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:03:31.055276 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-05a2ec6d-0ad2-4360-b09e-f55bd766e60e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:03:31.133722 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 761/1502] 10.222.0.50 () {58 vars in 958 bytes} [Tue Sep 13 16:03:30 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 139 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:03:31.147762 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccdcd381-c7b7-4063-95e6-e34442562a95 None None] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:03:31.148048 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccdcd381-c7b7-4063-95e6-e34442562a95 None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:03:31.148318 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccdcd381-c7b7-4063-95e6-e34442562a95 None None] PATH_INFO: `/` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:03:31.149630 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 742/1503] 10.222.0.50 () {54 vars in 964 bytes} [Tue Sep 13 16:03:31 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 16:03:31.155047 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1eb7a1aa-e63d-4142-a550-74b60d3c4b24 None None] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:03:31.155341 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1eb7a1aa-e63d-4142-a550-74b60d3c4b24 None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:03:31.155609 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1eb7a1aa-e63d-4142-a550-74b60d3c4b24 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:03:31.187134 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:03:31.237310 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.auth.core [None req-1eb7a1aa-e63d-4142-a550-74b60d3c4b24 None None] MFA Rules not processed for user `7fd6b5afd67b48f9b0f0597ccd4cf4df`. Rule list: `[]` (Enabled: `True`). {{(pid=81437) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:03:31.238434 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-1eb7a1aa-e63d-4142-a550-74b60d3c4b24 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:03:31.312362 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 762/1504] 10.222.0.50 () {58 vars in 1061 bytes} [Tue Sep 13 16:03:31 2022] POST /identity/v3/auth/tokens => generated 3120 bytes in 159 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:03:31.320035 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccdcd381-c7b7-4063-95e6-e34442562a95 None None] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:03:31.321647 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-ccdcd381-c7b7-4063-95e6-e34442562a95 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:03:31.371167 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:03:31.426432 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccdcd381-c7b7-4063-95e6-e34442562a95 None None] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:03:31.499885 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a8c7889-f44e-4e22-9f3f-e2cda8b9d62a service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7fd6b5afd67b48f9b0f0597ccd4cf4df', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:03:31.500410 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a8c7889-f44e-4e22-9f3f-e2cda8b9d62a service cinder] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:03:31.500685 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a8c7889-f44e-4e22-9f3f-e2cda8b9d62a service cinder] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:03:31.500949 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a8c7889-f44e-4e22-9f3f-e2cda8b9d62a service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:03:31.502514 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-5a8c7889-f44e-4e22-9f3f-e2cda8b9d62a service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:03:31.600847 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a8c7889-f44e-4e22-9f3f-e2cda8b9d62a service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:03:31.602265 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a8c7889-f44e-4e22-9f3f-e2cda8b9d62a service cinder] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:03:31.605747 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:03:31.621426 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 743/1505] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 16:03:31 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 302 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 16:03:31.667301 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7907d68-5fb9-4e16-8e1b-b2c759e38b52 None None] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:03:31.667593 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7907d68-5fb9-4e16-8e1b-b2c759e38b52 None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:03:31.667898 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7907d68-5fb9-4e16-8e1b-b2c759e38b52 None None] PATH_INFO: `/` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:03:31.669862 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 763/1506] 10.222.0.50 () {52 vars in 830 bytes} [Tue Sep 13 16:03:31 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 16:03:31.673485 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a8c7889-f44e-4e22-9f3f-e2cda8b9d62a service cinder] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:03:31.674682 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-5a8c7889-f44e-4e22-9f3f-e2cda8b9d62a service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:03:31.771983 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a8c7889-f44e-4e22-9f3f-e2cda8b9d62a service cinder] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:03:31.865253 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53772218-c1f5-472e-8fab-022bb5adb96e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:03:31.865962 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53772218-c1f5-472e-8fab-022bb5adb96e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:03:31.866147 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53772218-c1f5-472e-8fab-022bb5adb96e admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:03:31.866419 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53772218-c1f5-472e-8fab-022bb5adb96e admin admin] PATH_INFO: `/v3/users/337fe2ee2e244b7fa126b7280d1e4d60` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:03:31.879240 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53772218-c1f5-472e-8fab-022bb5adb96e admin admin] RBAC: Authorizing `identity:delete_user(user_id=337fe2ee2e244b7fa126b7280d1e4d60)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:03:31.880263 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: WARNING py.warnings [None req-53772218-c1f5-472e-8fab-022bb5adb96e admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:delete_user": "(role:admin and system_scope:all) or (role:admin and token.domain.id:%(target.user.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Sep 13 16:03:31.880263 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: warnings.warn(msg) Sep 13 16:03:31.880263 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: Sep 13 16:03:31.881467 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53772218-c1f5-472e-8fab-022bb5adb96e admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:03:31.890968 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:03:31.946581 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-53772218-c1f5-472e-8fab-022bb5adb96e admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '337fe2ee2e244b7fa126b7280d1e4d60'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:03:31.956184 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-53772218-c1f5-472e-8fab-022bb5adb96e admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '337fe2ee2e244b7fa126b7280d1e4d60'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:03:31.979816 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-53772218-c1f5-472e-8fab-022bb5adb96e admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '337fe2ee2e244b7fa126b7280d1e4d60'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:03:31.980265 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-53772218-c1f5-472e-8fab-022bb5adb96e admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '337fe2ee2e244b7fa126b7280d1e4d60'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:03:31.994545 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-53772218-c1f5-472e-8fab-022bb5adb96e admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '337fe2ee2e244b7fa126b7280d1e4d60'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:03:32.000112 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 744/1507] 10.222.0.50 () {58 vars in 1224 bytes} [Tue Sep 13 16:03:31 2022] DELETE /identity/v3/users/337fe2ee2e244b7fa126b7280d1e4d60 => generated 0 bytes in 327 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 16:03:32.004435 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec2be86e-b82c-41b1-a854-97f78f40544d None None] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:03:32.004916 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec2be86e-b82c-41b1-a854-97f78f40544d None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:03:32.005014 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec2be86e-b82c-41b1-a854-97f78f40544d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:03:32.063377 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.auth.core [None req-ec2be86e-b82c-41b1-a854-97f78f40544d None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81437) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:03:32.064385 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-ec2be86e-b82c-41b1-a854-97f78f40544d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:03:32.138086 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:03:32.150678 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 764/1508] 10.222.0.50 () {58 vars in 958 bytes} [Tue Sep 13 16:03:32 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 147 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:03:32.162541 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53772218-c1f5-472e-8fab-022bb5adb96e admin admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:03:32.163796 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-53772218-c1f5-472e-8fab-022bb5adb96e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:03:32.265895 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53772218-c1f5-472e-8fab-022bb5adb96e admin admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:03:32.320430 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:DELETE=10,keystone:SELECT=89,keystone:INSERT=1 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:03:32.341839 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91c5e461-34fb-49fb-801c-1596d07d18c1 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f436f07939d4855a21b195bc7566244', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:03:32.342447 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91c5e461-34fb-49fb-801c-1596d07d18c1 service neutron] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:03:32.342726 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91c5e461-34fb-49fb-801c-1596d07d18c1 service neutron] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:03:32.343058 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91c5e461-34fb-49fb-801c-1596d07d18c1 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:03:32.344629 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-91c5e461-34fb-49fb-801c-1596d07d18c1 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:03:32.451884 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91c5e461-34fb-49fb-801c-1596d07d18c1 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:03:32.453315 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91c5e461-34fb-49fb-801c-1596d07d18c1 service neutron] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:03:32.471482 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 745/1509] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 16:03:32 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 309 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 16:03:32.823783 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec2be86e-b82c-41b1-a854-97f78f40544d None None] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:03:32.826447 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-ec2be86e-b82c-41b1-a854-97f78f40544d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:03:32.838043 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec2be86e-b82c-41b1-a854-97f78f40544d None None] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:03:32.938618 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f24e253f-9de9-461d-aade-2cb71bb85b2f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:03:32.939276 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f24e253f-9de9-461d-aade-2cb71bb85b2f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:03:32.939550 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f24e253f-9de9-461d-aade-2cb71bb85b2f admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:03:32.939815 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f24e253f-9de9-461d-aade-2cb71bb85b2f admin admin] PATH_INFO: `/v3/projects/f20a7fb776934767baec85df294a0cc4` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:03:32.949421 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f24e253f-9de9-461d-aade-2cb71bb85b2f admin admin] RBAC: Authorizing `identity:delete_project(project_id=f20a7fb776934767baec85df294a0cc4)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:03:32.950475 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: WARNING py.warnings [None req-f24e253f-9de9-461d-aade-2cb71bb85b2f admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:delete_project": "(role:admin and system_scope:all) or (role:admin and domain_id:%(target.project.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Sep 13 16:03:32.950475 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: warnings.warn(msg) Sep 13 16:03:32.950475 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: Sep 13 16:03:32.951691 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f24e253f-9de9-461d-aade-2cb71bb85b2f admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:03:33.010365 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-f24e253f-9de9-461d-aade-2cb71bb85b2f admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'f20a7fb776934767baec85df294a0cc4', 'request_id': 'req-f24e253f-9de9-461d-aade-2cb71bb85b2f'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:03:33.020269 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-f24e253f-9de9-461d-aade-2cb71bb85b2f admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'f20a7fb776934767baec85df294a0cc4', 'request_id': 'req-f24e253f-9de9-461d-aade-2cb71bb85b2f'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:03:33.034942 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-f24e253f-9de9-461d-aade-2cb71bb85b2f admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'f20a7fb776934767baec85df294a0cc4', 'request_id': 'req-f24e253f-9de9-461d-aade-2cb71bb85b2f'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:03:33.057561 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-f24e253f-9de9-461d-aade-2cb71bb85b2f admin admin] The token cache is being invalidate because project f20a7fb776934767baec85df294a0cc4 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81437) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 13 16:03:33.057850 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-f24e253f-9de9-461d-aade-2cb71bb85b2f admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:03:33.058278 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-f24e253f-9de9-461d-aade-2cb71bb85b2f admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:03:33.060645 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 765/1510] 10.222.0.50 () {58 vars in 1233 bytes} [Tue Sep 13 16:03:32 2022] DELETE /identity/v3/projects/f20a7fb776934767baec85df294a0cc4 => generated 0 bytes in 237 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 16:03:42.469232 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=62 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:03:43.055510 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=84,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:04:56.526508 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15fcac96-45aa-4356-b6c4-0fe78e6fd7a5 None None] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:04:56.527131 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15fcac96-45aa-4356-b6c4-0fe78e6fd7a5 None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:04:56.527131 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15fcac96-45aa-4356-b6c4-0fe78e6fd7a5 None None] PATH_INFO: `/` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:04:56.528464 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 746/1511] 127.0.0.1 () {54 vars in 901 bytes} [Tue Sep 13 16:04:56 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 16:04:56.533485 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43ca4ff3-4de0-4c46-9a40-2250ab2b495f None None] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:04:56.533906 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43ca4ff3-4de0-4c46-9a40-2250ab2b495f None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:04:56.534031 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43ca4ff3-4de0-4c46-9a40-2250ab2b495f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:04:56.614031 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.auth.core [None req-43ca4ff3-4de0-4c46-9a40-2250ab2b495f None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81437) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:04:56.615265 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-43ca4ff3-4de0-4c46-9a40-2250ab2b495f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:04:56.717216 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 766/1512] 127.0.0.1 () {58 vars in 998 bytes} [Tue Sep 13 16:04:56 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 185 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:04:56.870852 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15fcac96-45aa-4356-b6c4-0fe78e6fd7a5 None None] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:04:56.873041 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-15fcac96-45aa-4356-b6c4-0fe78e6fd7a5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:04:56.990605 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15fcac96-45aa-4356-b6c4-0fe78e6fd7a5 None None] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:04:56.996230 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99503f21-3431-43e9-858e-c57ca6a0cf98 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4e039b46c5e3445bbb39904b00c5430e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'admin', 'ResellerAdmin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:04:56.996829 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99503f21-3431-43e9-858e-c57ca6a0cf98 service nova] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:04:56.997103 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99503f21-3431-43e9-858e-c57ca6a0cf98 service nova] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:04:56.997371 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99503f21-3431-43e9-858e-c57ca6a0cf98 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:04:57.005355 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99503f21-3431-43e9-858e-c57ca6a0cf98 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:04:57.006985 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99503f21-3431-43e9-858e-c57ca6a0cf98 service nova] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:04:57.016875 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 747/1513] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 16:04:56 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 146 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 16:04:58.104830 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5d8d564-656e-42a9-bf42-06efa7c3765d None None] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:04:58.105358 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5d8d564-656e-42a9-bf42-06efa7c3765d None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:04:58.105465 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5d8d564-656e-42a9-bf42-06efa7c3765d None None] PATH_INFO: `/` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:04:58.106728 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 767/1514] 127.0.0.1 () {54 vars in 901 bytes} [Tue Sep 13 16:04:58 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 16:04:58.113030 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2781c8cf-271c-4b29-8ca0-e7462da5ab73 None None] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:04:58.113505 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2781c8cf-271c-4b29-8ca0-e7462da5ab73 None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:04:58.113607 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2781c8cf-271c-4b29-8ca0-e7462da5ab73 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:04:58.134724 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.auth.core [None req-2781c8cf-271c-4b29-8ca0-e7462da5ab73 None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81438) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:04:58.135965 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-2781c8cf-271c-4b29-8ca0-e7462da5ab73 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:04:58.143883 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 748/1515] 127.0.0.1 () {58 vars in 998 bytes} [Tue Sep 13 16:04:58 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:04:58.294828 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5d8d564-656e-42a9-bf42-06efa7c3765d None None] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:04:58.309937 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5d8d564-656e-42a9-bf42-06efa7c3765d None None] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:04:58.315457 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca34094c-08dc-4534-ab8b-42d1b11ad2e4 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4e039b46c5e3445bbb39904b00c5430e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'admin', 'ResellerAdmin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:04:58.316056 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca34094c-08dc-4534-ab8b-42d1b11ad2e4 service nova] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:04:58.316366 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca34094c-08dc-4534-ab8b-42d1b11ad2e4 service nova] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:04:58.316638 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca34094c-08dc-4534-ab8b-42d1b11ad2e4 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:04:58.325812 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca34094c-08dc-4534-ab8b-42d1b11ad2e4 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:04:58.327444 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca34094c-08dc-4534-ab8b-42d1b11ad2e4 service nova] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:04:58.336972 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 768/1516] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 16:04:58 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 43 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 16:04:58.511194 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29e3a534-6b33-4e3b-bd0f-02d76d71e67b None None] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:04:58.511390 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29e3a534-6b33-4e3b-bd0f-02d76d71e67b None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:04:58.511662 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29e3a534-6b33-4e3b-bd0f-02d76d71e67b None None] PATH_INFO: `/v3` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:04:58.512923 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 749/1517] 10.222.0.59 () {54 vars in 925 bytes} [Tue Sep 13 16:04:58 2022] GET /identity/v3 => generated 255 bytes in 3 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 16:04:58.519845 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da28285f-dd3b-4764-b890-f55e9be7f37f None None] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:04:58.520137 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da28285f-dd3b-4764-b890-f55e9be7f37f None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:04:58.520676 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da28285f-dd3b-4764-b890-f55e9be7f37f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:04:58.580363 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.auth.core [None req-da28285f-dd3b-4764-b890-f55e9be7f37f None None] MFA Rules not processed for user `31480b61418340bd88583f034f67412e`. Rule list: `[]` (Enabled: `True`). {{(pid=81437) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:04:58.581571 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-da28285f-dd3b-4764-b890-f55e9be7f37f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:04:58.597924 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:04:58.655030 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 769/1518] 10.222.0.59 () {58 vars in 1015 bytes} [Tue Sep 13 16:04:58 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 136 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:05:02.143789 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0c43e33-a3ba-499c-a975-e1afb22d0740 None None] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:05:02.144201 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0c43e33-a3ba-499c-a975-e1afb22d0740 None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:05:02.144323 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0c43e33-a3ba-499c-a975-e1afb22d0740 None None] PATH_INFO: `/` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:05:02.145909 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 750/1519] 10.222.0.50 () {54 vars in 961 bytes} [Tue Sep 13 16:05:02 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 16:05:02.151760 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eecd6387-eb27-42d6-94b4-ef76ca53b196 None None] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:05:02.152219 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eecd6387-eb27-42d6-94b4-ef76ca53b196 None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:05:02.152332 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eecd6387-eb27-42d6-94b4-ef76ca53b196 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:05:02.172249 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.auth.core [None req-eecd6387-eb27-42d6-94b4-ef76ca53b196 None None] MFA Rules not processed for user `31480b61418340bd88583f034f67412e`. Rule list: `[]` (Enabled: `True`). {{(pid=81437) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:05:02.173450 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-eecd6387-eb27-42d6-94b4-ef76ca53b196 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:05:02.181953 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 770/1520] 10.222.0.50 () {58 vars in 1058 bytes} [Tue Sep 13 16:05:02 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:05:02.189675 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0c43e33-a3ba-499c-a975-e1afb22d0740 None None] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:05:02.203735 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0c43e33-a3ba-499c-a975-e1afb22d0740 None None] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:05:02.208674 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2d4585d-5aed-4ecb-8dfe-1d50e1993b92 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '31480b61418340bd88583f034f67412e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:05:02.209263 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2d4585d-5aed-4ecb-8dfe-1d50e1993b92 service placement] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:05:02.209545 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2d4585d-5aed-4ecb-8dfe-1d50e1993b92 service placement] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:05:02.209817 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2d4585d-5aed-4ecb-8dfe-1d50e1993b92 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:05:02.218631 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2d4585d-5aed-4ecb-8dfe-1d50e1993b92 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:05:02.220262 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2d4585d-5aed-4ecb-8dfe-1d50e1993b92 service placement] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:05:02.229935 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 751/1521] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 16:05:02 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 41 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 16:05:09.831339 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6440733c-29dd-48a4-b112-425af975c7bf None None] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:05:09.831925 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6440733c-29dd-48a4-b112-425af975c7bf None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:05:09.831925 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6440733c-29dd-48a4-b112-425af975c7bf None None] PATH_INFO: `/` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:05:09.833229 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 771/1522] 127.0.0.1 () {54 vars in 901 bytes} [Tue Sep 13 16:05:09 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 16:05:09.840947 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b4ce2ea-3f81-4f84-9cdf-9f8a0fad805b None None] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:05:09.841665 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b4ce2ea-3f81-4f84-9cdf-9f8a0fad805b None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:05:09.841665 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b4ce2ea-3f81-4f84-9cdf-9f8a0fad805b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:05:09.861542 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.auth.core [None req-8b4ce2ea-3f81-4f84-9cdf-9f8a0fad805b None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81438) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:05:09.862733 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-8b4ce2ea-3f81-4f84-9cdf-9f8a0fad805b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:05:09.871271 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 752/1523] 127.0.0.1 () {58 vars in 998 bytes} [Tue Sep 13 16:05:09 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:05:10.024545 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6440733c-29dd-48a4-b112-425af975c7bf None None] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:05:10.039504 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6440733c-29dd-48a4-b112-425af975c7bf None None] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:05:10.045125 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-619e50e5-90cc-4f5c-947a-6985377509ac service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4e039b46c5e3445bbb39904b00c5430e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'admin', 'ResellerAdmin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:05:10.045722 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-619e50e5-90cc-4f5c-947a-6985377509ac service nova] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:05:10.046007 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-619e50e5-90cc-4f5c-947a-6985377509ac service nova] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:05:10.046279 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-619e50e5-90cc-4f5c-947a-6985377509ac service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:05:10.054740 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-619e50e5-90cc-4f5c-947a-6985377509ac service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:05:10.056431 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-619e50e5-90cc-4f5c-947a-6985377509ac service nova] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:05:10.065894 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 772/1524] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 16:05:10 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 42 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 16:05:11.123574 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d590e8d0-7646-4fe6-bf86-65a9f79c7390 None None] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:05:11.124078 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d590e8d0-7646-4fe6-bf86-65a9f79c7390 None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:05:11.124173 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d590e8d0-7646-4fe6-bf86-65a9f79c7390 None None] PATH_INFO: `/` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:05:11.125477 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 753/1525] 127.0.0.1 () {54 vars in 901 bytes} [Tue Sep 13 16:05:11 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 16:05:11.131440 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d77fd256-435d-4a95-af8a-6699d3232a05 None None] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:05:11.131822 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d77fd256-435d-4a95-af8a-6699d3232a05 None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:05:11.131991 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d77fd256-435d-4a95-af8a-6699d3232a05 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:05:11.152536 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.auth.core [None req-d77fd256-435d-4a95-af8a-6699d3232a05 None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81437) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:05:11.153716 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-d77fd256-435d-4a95-af8a-6699d3232a05 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:05:11.161309 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 773/1526] 127.0.0.1 () {58 vars in 998 bytes} [Tue Sep 13 16:05:11 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:05:11.311179 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d590e8d0-7646-4fe6-bf86-65a9f79c7390 None None] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:05:11.326195 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d590e8d0-7646-4fe6-bf86-65a9f79c7390 None None] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:05:11.331646 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5afdee1-f429-4dbf-9dd0-988127510d13 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4e039b46c5e3445bbb39904b00c5430e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'admin', 'ResellerAdmin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:05:11.332242 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5afdee1-f429-4dbf-9dd0-988127510d13 service nova] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:05:11.332522 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5afdee1-f429-4dbf-9dd0-988127510d13 service nova] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:05:11.332797 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5afdee1-f429-4dbf-9dd0-988127510d13 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:05:11.341573 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5afdee1-f429-4dbf-9dd0-988127510d13 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:05:11.343182 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5afdee1-f429-4dbf-9dd0-988127510d13 service nova] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:05:11.352582 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 754/1527] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 16:05:11 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 42 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 16:05:15.395978 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fa90d10-b758-4cef-87f7-b9bf72e45edc None None] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:05:15.396518 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fa90d10-b758-4cef-87f7-b9bf72e45edc None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:05:15.396518 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fa90d10-b758-4cef-87f7-b9bf72e45edc None None] PATH_INFO: `/` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:05:15.398708 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 774/1528] 127.0.0.1 () {54 vars in 917 bytes} [Tue Sep 13 16:05:15 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 16:05:15.403824 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5915a13-0b0d-40d8-aaba-42584fecf521 None None] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:05:15.404267 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5915a13-0b0d-40d8-aaba-42584fecf521 None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:05:15.404364 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5915a13-0b0d-40d8-aaba-42584fecf521 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:05:15.424149 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.auth.core [None req-a5915a13-0b0d-40d8-aaba-42584fecf521 None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81438) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:05:15.425346 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-a5915a13-0b0d-40d8-aaba-42584fecf521 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:05:15.433245 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 755/1529] 127.0.0.1 () {58 vars in 1014 bytes} [Tue Sep 13 16:05:15 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:05:15.720817 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fa90d10-b758-4cef-87f7-b9bf72e45edc None None] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:05:15.722497 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-9fa90d10-b758-4cef-87f7-b9bf72e45edc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:05:15.799785 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fa90d10-b758-4cef-87f7-b9bf72e45edc None None] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:05:15.803313 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b46e3015-671e-4cd8-aeb7-2c5b112faaef service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f436f07939d4855a21b195bc7566244', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:05:15.804033 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b46e3015-671e-4cd8-aeb7-2c5b112faaef service neutron] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:05:15.804309 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b46e3015-671e-4cd8-aeb7-2c5b112faaef service neutron] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:05:15.804583 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b46e3015-671e-4cd8-aeb7-2c5b112faaef service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:05:15.813431 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b46e3015-671e-4cd8-aeb7-2c5b112faaef service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:05:15.814860 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b46e3015-671e-4cd8-aeb7-2c5b112faaef service neutron] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:05:15.824772 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 775/1530] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 16:05:15 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 104 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 16:05:17.996866 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30c3ca96-f4a1-47aa-afda-1a36f726a1fa None None] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:05:17.997415 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30c3ca96-f4a1-47aa-afda-1a36f726a1fa None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:05:17.997415 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30c3ca96-f4a1-47aa-afda-1a36f726a1fa None None] PATH_INFO: `/` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:05:17.998712 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 756/1531] 127.0.0.1 () {54 vars in 917 bytes} [Tue Sep 13 16:05:17 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 16:05:18.003842 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0469a90a-4674-41d6-87d2-1f6108e2b9eb None None] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:05:18.004303 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0469a90a-4674-41d6-87d2-1f6108e2b9eb None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:05:18.004406 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0469a90a-4674-41d6-87d2-1f6108e2b9eb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:05:18.024440 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.auth.core [None req-0469a90a-4674-41d6-87d2-1f6108e2b9eb None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81437) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:05:18.025639 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-0469a90a-4674-41d6-87d2-1f6108e2b9eb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:05:18.033292 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 776/1532] 127.0.0.1 () {58 vars in 1014 bytes} [Tue Sep 13 16:05:18 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:05:18.318615 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30c3ca96-f4a1-47aa-afda-1a36f726a1fa None None] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:05:18.329866 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30c3ca96-f4a1-47aa-afda-1a36f726a1fa None None] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:05:18.333331 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-608a0c90-265a-4695-a1e4-e70ddd58c49d service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f436f07939d4855a21b195bc7566244', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:05:18.333948 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-608a0c90-265a-4695-a1e4-e70ddd58c49d service neutron] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:05:18.334254 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-608a0c90-265a-4695-a1e4-e70ddd58c49d service neutron] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:05:18.334529 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-608a0c90-265a-4695-a1e4-e70ddd58c49d service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:05:18.343654 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-608a0c90-265a-4695-a1e4-e70ddd58c49d service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:05:18.345066 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-608a0c90-265a-4695-a1e4-e70ddd58c49d service neutron] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:05:18.354819 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 757/1533] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 16:05:18 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 37 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 16:05:22.102438 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dd4ef63-2875-4c1a-8c9f-1f3ba72e3634 None None] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:05:22.102968 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dd4ef63-2875-4c1a-8c9f-1f3ba72e3634 None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:05:22.103080 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dd4ef63-2875-4c1a-8c9f-1f3ba72e3634 None None] PATH_INFO: `/` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:05:22.104411 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 777/1534] 127.0.0.1 () {54 vars in 917 bytes} [Tue Sep 13 16:05:22 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 16:05:22.109568 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e64b7ba0-74f2-4bfc-b3e4-efe276e72cf0 None None] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:05:22.110002 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e64b7ba0-74f2-4bfc-b3e4-efe276e72cf0 None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:05:22.110112 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e64b7ba0-74f2-4bfc-b3e4-efe276e72cf0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:05:22.129969 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.auth.core [None req-e64b7ba0-74f2-4bfc-b3e4-efe276e72cf0 None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81438) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:05:22.131785 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-e64b7ba0-74f2-4bfc-b3e4-efe276e72cf0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:05:22.139770 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 758/1535] 127.0.0.1 () {58 vars in 1014 bytes} [Tue Sep 13 16:05:22 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:05:22.446057 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1dd4ef63-2875-4c1a-8c9f-1f3ba72e3634 None None] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:05:22.447781 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-1dd4ef63-2875-4c1a-8c9f-1f3ba72e3634 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:05:22.511769 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:05:22.525317 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1dd4ef63-2875-4c1a-8c9f-1f3ba72e3634 None None] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:05:22.528935 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba531a7a-438a-4e87-89ab-63d1625d7285 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7fd6b5afd67b48f9b0f0597ccd4cf4df', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:05:22.529541 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba531a7a-438a-4e87-89ab-63d1625d7285 service cinder] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:05:22.529819 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba531a7a-438a-4e87-89ab-63d1625d7285 service cinder] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:05:22.530093 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba531a7a-438a-4e87-89ab-63d1625d7285 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:05:22.539486 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba531a7a-438a-4e87-89ab-63d1625d7285 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:05:22.540907 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba531a7a-438a-4e87-89ab-63d1625d7285 service cinder] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:05:22.550167 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 778/1536] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 16:05:22 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 105 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 16:05:24.667441 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbd055e5-21fd-438c-88b0-f9a63b4a04ad None None] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:05:24.668019 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbd055e5-21fd-438c-88b0-f9a63b4a04ad None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:05:24.668019 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbd055e5-21fd-438c-88b0-f9a63b4a04ad None None] PATH_INFO: `/` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:05:24.669300 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 759/1537] 127.0.0.1 () {54 vars in 917 bytes} [Tue Sep 13 16:05:24 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 16:05:24.674628 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49455ff8-8253-4be0-9026-2d198d78af33 None None] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:05:24.675043 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49455ff8-8253-4be0-9026-2d198d78af33 None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:05:24.675204 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49455ff8-8253-4be0-9026-2d198d78af33 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:05:24.694959 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.auth.core [None req-49455ff8-8253-4be0-9026-2d198d78af33 None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81437) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:05:24.696146 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-49455ff8-8253-4be0-9026-2d198d78af33 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:05:24.703745 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 779/1538] 127.0.0.1 () {58 vars in 1014 bytes} [Tue Sep 13 16:05:24 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:05:25.002118 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbd055e5-21fd-438c-88b0-f9a63b4a04ad None None] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:05:25.003858 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-bbd055e5-21fd-438c-88b0-f9a63b4a04ad None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:05:25.012211 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=84 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:05:25.018786 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbd055e5-21fd-438c-88b0-f9a63b4a04ad None None] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:05:25.022419 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61afa2c5-c339-4a4e-9957-7cb2b43499c8 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7fd6b5afd67b48f9b0f0597ccd4cf4df', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:05:25.023045 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61afa2c5-c339-4a4e-9957-7cb2b43499c8 service cinder] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:05:25.023326 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61afa2c5-c339-4a4e-9957-7cb2b43499c8 service cinder] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:05:25.023601 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61afa2c5-c339-4a4e-9957-7cb2b43499c8 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:05:25.032382 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61afa2c5-c339-4a4e-9957-7cb2b43499c8 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:05:25.033785 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61afa2c5-c339-4a4e-9957-7cb2b43499c8 service cinder] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:05:25.043621 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 760/1539] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 16:05:25 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 42 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 16:05:34.689559 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=11 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:05:35.038980 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=5 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:06:47.779364 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3049045-769c-439f-88c0-27d719a981f0 None None] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:06:47.780723 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3049045-769c-439f-88c0-27d719a981f0 None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:06:47.780723 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3049045-769c-439f-88c0-27d719a981f0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:06:47.813865 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.auth.core [None req-e3049045-769c-439f-88c0-27d719a981f0 None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81437) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:06:47.815125 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-e3049045-769c-439f-88c0-27d719a981f0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:06:47.831201 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 780/1540] 10.222.0.50 () {58 vars in 958 bytes} [Tue Sep 13 16:06:47 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 53 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 13 16:06:47.840117 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3da0cb7e-a25b-4033-a0f1-7a254b72fdc2 None None] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:06:47.841499 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3da0cb7e-a25b-4033-a0f1-7a254b72fdc2 None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:06:47.841499 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3da0cb7e-a25b-4033-a0f1-7a254b72fdc2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:06:47.861183 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbd06099-5438-4e6d-b521-5bab2c0b1141 None None] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:06:47.861528 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbd06099-5438-4e6d-b521-5bab2c0b1141 None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:06:47.861744 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbd06099-5438-4e6d-b521-5bab2c0b1141 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:06:47.885436 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.auth.core [None req-3da0cb7e-a25b-4033-a0f1-7a254b72fdc2 None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81438) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:06:47.886678 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-3da0cb7e-a25b-4033-a0f1-7a254b72fdc2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:06:47.910856 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.auth.core [None req-bbd06099-5438-4e6d-b521-5bab2c0b1141 None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81437) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:06:47.912100 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-bbd06099-5438-4e6d-b521-5bab2c0b1141 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:06:47.915549 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 761/1541] 10.222.0.50 () {58 vars in 958 bytes} [Tue Sep 13 16:06:47 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 76 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:06:47.927759 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3da0cb7e-a25b-4033-a0f1-7a254b72fdc2 None None] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:06:47.935369 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 781/1542] 10.222.0.50 () {58 vars in 958 bytes} [Tue Sep 13 16:06:47 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 75 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:06:47.942782 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f7dd5ef-8416-43a4-ac49-fbb7d45235b0 None None] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:06:47.943131 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f7dd5ef-8416-43a4-ac49-fbb7d45235b0 None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:06:47.943376 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f7dd5ef-8416-43a4-ac49-fbb7d45235b0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:06:47.948223 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3da0cb7e-a25b-4033-a0f1-7a254b72fdc2 None None] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:06:47.960280 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb0b0cbd-3dd1-406f-b757-f2f1f3d25450 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:06:47.961547 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb0b0cbd-3dd1-406f-b757-f2f1f3d25450 admin admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:06:47.962148 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb0b0cbd-3dd1-406f-b757-f2f1f3d25450 admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:06:47.962653 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb0b0cbd-3dd1-406f-b757-f2f1f3d25450 admin admin] PATH_INFO: `/v3/domains` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:06:47.963604 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb0b0cbd-3dd1-406f-b757-f2f1f3d25450 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81438) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 16:06:47.964179 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb0b0cbd-3dd1-406f-b757-f2f1f3d25450 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:06:47.965740 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: WARNING py.warnings [None req-cb0b0cbd-3dd1-406f-b757-f2f1f3d25450 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_domains": "role:reader and system_scope:all" failed scope check. The token used to make the request was project scoped but the policy requires ['system'] scope. This behavior may change in the future where using the intended scope is required Sep 13 16:06:47.965740 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: warnings.warn(msg) Sep 13 16:06:47.965740 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: Sep 13 16:06:47.967664 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb0b0cbd-3dd1-406f-b757-f2f1f3d25450 admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:06:47.988182 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 762/1543] 10.222.0.50 () {58 vars in 1153 bytes} [Tue Sep 13 16:06:47 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 61 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 13 16:06:47.990325 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.auth.core [None req-2f7dd5ef-8416-43a4-ac49-fbb7d45235b0 None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81437) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:06:47.991781 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-2f7dd5ef-8416-43a4-ac49-fbb7d45235b0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:06:48.000086 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb0b0cbd-3dd1-406f-b757-f2f1f3d25450 admin admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:06:48.013430 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 782/1544] 10.222.0.50 () {58 vars in 958 bytes} [Tue Sep 13 16:06:47 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 71 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:06:48.017084 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f7dd5ef-8416-43a4-ac49-fbb7d45235b0 None None] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:06:48.017694 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb0b0cbd-3dd1-406f-b757-f2f1f3d25450 admin admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:06:48.023597 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f0b51cd-f8ea-427c-a9b7-649c0651e6ea admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:06:48.024710 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f0b51cd-f8ea-427c-a9b7-649c0651e6ea admin admin] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:06:48.025237 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f0b51cd-f8ea-427c-a9b7-649c0651e6ea admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:06:48.025845 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f0b51cd-f8ea-427c-a9b7-649c0651e6ea admin admin] PATH_INFO: `/v3/projects` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:06:48.026861 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f0b51cd-f8ea-427c-a9b7-649c0651e6ea admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:06:48.036689 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f0b51cd-f8ea-427c-a9b7-649c0651e6ea admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:06:48.038039 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f7dd5ef-8416-43a4-ac49-fbb7d45235b0 None None] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:06:48.091478 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbc8fe4a-46ed-4320-944b-e1a62961c1d5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:06:48.092726 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbc8fe4a-46ed-4320-944b-e1a62961c1d5 admin admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:06:48.093049 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbc8fe4a-46ed-4320-944b-e1a62961c1d5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:06:48.093477 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbc8fe4a-46ed-4320-944b-e1a62961c1d5 admin admin] PATH_INFO: `/v3/domains` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:06:48.094457 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbc8fe4a-46ed-4320-944b-e1a62961c1d5 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81437) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 16:06:48.094790 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbc8fe4a-46ed-4320-944b-e1a62961c1d5 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:06:48.096682 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbc8fe4a-46ed-4320-944b-e1a62961c1d5 admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:06:48.133360 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 783/1545] 10.222.0.50 () {58 vars in 1153 bytes} [Tue Sep 13 16:06:48 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 117 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 13 16:06:48.140279 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96608a66-80af-4aaf-a2d9-7f4f77647ee0 None None] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:06:48.140772 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96608a66-80af-4aaf-a2d9-7f4f77647ee0 None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:06:48.141063 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96608a66-80af-4aaf-a2d9-7f4f77647ee0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:06:48.154380 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 763/1546] 10.222.0.50 () {60 vars in 1153 bytes} [Tue Sep 13 16:06:47 2022] POST /identity/v3/projects => generated 373 bytes in 155 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 13 16:06:48.155521 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f0b51cd-f8ea-427c-a9b7-649c0651e6ea admin admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:06:48.184100 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.auth.core [None req-96608a66-80af-4aaf-a2d9-7f4f77647ee0 None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81437) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:06:48.185737 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-96608a66-80af-4aaf-a2d9-7f4f77647ee0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:06:48.274508 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f0b51cd-f8ea-427c-a9b7-649c0651e6ea admin admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:06:48.286523 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b11250fe-80e0-4eca-8edd-d86ddf95d645 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:06:48.287097 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b11250fe-80e0-4eca-8edd-d86ddf95d645 admin admin] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:06:48.287364 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b11250fe-80e0-4eca-8edd-d86ddf95d645 admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:06:48.287636 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b11250fe-80e0-4eca-8edd-d86ddf95d645 admin admin] PATH_INFO: `/v3/projects` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:06:48.288265 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b11250fe-80e0-4eca-8edd-d86ddf95d645 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:06:48.290166 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b11250fe-80e0-4eca-8edd-d86ddf95d645 admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:06:48.318424 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 784/1547] 10.222.0.50 () {58 vars in 958 bytes} [Tue Sep 13 16:06:48 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 184 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:06:48.319778 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96608a66-80af-4aaf-a2d9-7f4f77647ee0 None None] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:06:48.333845 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96608a66-80af-4aaf-a2d9-7f4f77647ee0 None None] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:06:48.340335 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91bd160e-4de5-4c01-80dd-85c75001fff2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:06:48.341519 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91bd160e-4de5-4c01-80dd-85c75001fff2 admin admin] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:06:48.342029 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91bd160e-4de5-4c01-80dd-85c75001fff2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:06:48.342759 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91bd160e-4de5-4c01-80dd-85c75001fff2 admin admin] PATH_INFO: `/v3/users` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:06:48.343755 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91bd160e-4de5-4c01-80dd-85c75001fff2 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:06:48.345822 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 764/1548] 10.222.0.50 () {60 vars in 1153 bytes} [Tue Sep 13 16:06:48 2022] POST /identity/v3/projects => generated 405 bytes in 191 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 16:06:48.346298 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91bd160e-4de5-4c01-80dd-85c75001fff2 admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:06:48.347187 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecdc7c6b-63ef-4891-890c-1a1a28ec9611 None None] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:06:48.347474 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecdc7c6b-63ef-4891-890c-1a1a28ec9611 None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:06:48.347745 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecdc7c6b-63ef-4891-890c-1a1a28ec9611 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:06:48.381884 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.auth.core [None req-ecdc7c6b-63ef-4891-890c-1a1a28ec9611 None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81438) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:06:48.383696 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-ecdc7c6b-63ef-4891-890c-1a1a28ec9611 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:06:48.387476 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 785/1549] 10.222.0.50 () {60 vars in 1144 bytes} [Tue Sep 13 16:06:48 2022] POST /identity/v3/users => generated 287 bytes in 68 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 16:06:48.389556 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd305a28-ca4d-49e2-b463-df1b0722f80d None None] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:06:48.389930 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd305a28-ca4d-49e2-b463-df1b0722f80d None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:06:48.390398 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd305a28-ca4d-49e2-b463-df1b0722f80d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:06:48.424105 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.auth.core [None req-fd305a28-ca4d-49e2-b463-df1b0722f80d None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81437) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:06:48.425710 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-fd305a28-ca4d-49e2-b463-df1b0722f80d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:06:48.458089 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 765/1550] 10.222.0.50 () {58 vars in 958 bytes} [Tue Sep 13 16:06:48 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 112 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:06:48.458934 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecdc7c6b-63ef-4891-890c-1a1a28ec9611 None None] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:06:48.470967 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecdc7c6b-63ef-4891-890c-1a1a28ec9611 None None] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:06:48.475673 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83f98a12-5d49-46f1-b71f-240cf455751d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:06:48.476273 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83f98a12-5d49-46f1-b71f-240cf455751d admin admin] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:06:48.476554 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83f98a12-5d49-46f1-b71f-240cf455751d admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:06:48.476824 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83f98a12-5d49-46f1-b71f-240cf455751d admin admin] PATH_INFO: `/v3/users` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:06:48.477448 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83f98a12-5d49-46f1-b71f-240cf455751d admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:06:48.478463 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: WARNING py.warnings [None req-83f98a12-5d49-46f1-b71f-240cf455751d admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_user": "(role:admin and system_scope:all) or (role:admin and token.domain.id:%(target.user.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Sep 13 16:06:48.478463 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: warnings.warn(msg) Sep 13 16:06:48.478463 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: Sep 13 16:06:48.479730 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83f98a12-5d49-46f1-b71f-240cf455751d admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:06:48.488288 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 786/1551] 10.222.0.50 () {58 vars in 958 bytes} [Tue Sep 13 16:06:48 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:06:48.489530 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9fdce94-d97e-40d8-87ef-6da46c14b3b3 None None] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:06:48.489808 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9fdce94-d97e-40d8-87ef-6da46c14b3b3 None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:06:48.490082 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9fdce94-d97e-40d8-87ef-6da46c14b3b3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:06:48.508515 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 766/1552] 10.222.0.50 () {60 vars in 1144 bytes} [Tue Sep 13 16:06:48 2022] POST /identity/v3/users => generated 303 bytes in 47 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 16:06:48.508515 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-218b0330-a6c4-4feb-af10-f29bbb7d6395 None None] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:06:48.508515 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-218b0330-a6c4-4feb-af10-f29bbb7d6395 None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:06:48.508515 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-218b0330-a6c4-4feb-af10-f29bbb7d6395 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:06:48.510862 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.auth.core [None req-e9fdce94-d97e-40d8-87ef-6da46c14b3b3 None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81437) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:06:48.512088 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-e9fdce94-d97e-40d8-87ef-6da46c14b3b3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:06:48.520712 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 787/1553] 10.222.0.50 () {58 vars in 958 bytes} [Tue Sep 13 16:06:48 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:06:48.521553 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9fdce94-d97e-40d8-87ef-6da46c14b3b3 None None] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:06:48.531065 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9fdce94-d97e-40d8-87ef-6da46c14b3b3 None None] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:06:48.533501 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.auth.core [None req-218b0330-a6c4-4feb-af10-f29bbb7d6395 None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81438) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:06:48.534701 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-218b0330-a6c4-4feb-af10-f29bbb7d6395 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:06:48.535670 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64985646-27cf-4589-906f-b41572c77940 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:06:48.536229 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64985646-27cf-4589-906f-b41572c77940 admin admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:06:48.536620 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64985646-27cf-4589-906f-b41572c77940 admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:06:48.536779 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64985646-27cf-4589-906f-b41572c77940 admin admin] PATH_INFO: `/v3/roles` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:06:48.537408 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64985646-27cf-4589-906f-b41572c77940 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:06:48.539038 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64985646-27cf-4589-906f-b41572c77940 admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:06:48.542664 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 767/1554] 10.222.0.50 () {58 vars in 958 bytes} [Tue Sep 13 16:06:48 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 37 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:06:48.543942 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a62e59e-8e86-408b-b089-46cbe056d731 None None] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:06:48.544226 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a62e59e-8e86-408b-b089-46cbe056d731 None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:06:48.544497 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a62e59e-8e86-408b-b089-46cbe056d731 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:06:48.552002 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 788/1555] 10.222.0.50 () {58 vars in 1122 bytes} [Tue Sep 13 16:06:48 2022] GET /identity/v3/roles => generated 1420 bytes in 28 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 16:06:48.552002 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2a9d05f-75b1-4cd2-a5d7-44247dcb57fa None None] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:06:48.552002 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2a9d05f-75b1-4cd2-a5d7-44247dcb57fa None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:06:48.552002 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2a9d05f-75b1-4cd2-a5d7-44247dcb57fa None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:06:48.566920 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.auth.core [None req-8a62e59e-8e86-408b-b089-46cbe056d731 None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81438) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:06:48.568161 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-8a62e59e-8e86-408b-b089-46cbe056d731 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:06:48.571655 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.auth.core [None req-d2a9d05f-75b1-4cd2-a5d7-44247dcb57fa None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81437) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:06:48.572904 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-d2a9d05f-75b1-4cd2-a5d7-44247dcb57fa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:06:48.577192 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 768/1556] 10.222.0.50 () {58 vars in 958 bytes} [Tue Sep 13 16:06:48 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:06:48.578371 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16b6e417-0ec4-4a40-853b-e969ccd1c578 None None] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:06:48.578709 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16b6e417-0ec4-4a40-853b-e969ccd1c578 None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:06:48.578982 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16b6e417-0ec4-4a40-853b-e969ccd1c578 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:06:48.582583 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 789/1557] 10.222.0.50 () {58 vars in 958 bytes} [Tue Sep 13 16:06:48 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:06:48.587945 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2a9d05f-75b1-4cd2-a5d7-44247dcb57fa None None] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:06:48.594664 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:06:48.598090 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2a9d05f-75b1-4cd2-a5d7-44247dcb57fa None None] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:06:48.603402 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05c1dafe-ad46-4b11-b9db-14ffc083060e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:06:48.604014 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05c1dafe-ad46-4b11-b9db-14ffc083060e admin admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:06:48.604301 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05c1dafe-ad46-4b11-b9db-14ffc083060e admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:06:48.604565 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05c1dafe-ad46-4b11-b9db-14ffc083060e admin admin] PATH_INFO: `/v3/domains` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:06:48.605911 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05c1dafe-ad46-4b11-b9db-14ffc083060e admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81437) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 16:06:48.605911 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05c1dafe-ad46-4b11-b9db-14ffc083060e admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:06:48.607289 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05c1dafe-ad46-4b11-b9db-14ffc083060e admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:06:48.609050 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.auth.core [None req-16b6e417-0ec4-4a40-853b-e969ccd1c578 None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81438) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:06:48.610351 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-16b6e417-0ec4-4a40-853b-e969ccd1c578 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:06:48.618800 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 769/1558] 10.222.0.50 () {58 vars in 958 bytes} [Tue Sep 13 16:06:48 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 41 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:06:48.619513 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 790/1559] 10.222.0.50 () {58 vars in 1153 bytes} [Tue Sep 13 16:06:48 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 16:06:48.619996 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16b6e417-0ec4-4a40-853b-e969ccd1c578 None None] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:06:48.620900 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42fecae6-b743-4dce-b0dd-f1edee2cdb0a None None] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:06:48.621238 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42fecae6-b743-4dce-b0dd-f1edee2cdb0a None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:06:48.621569 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42fecae6-b743-4dce-b0dd-f1edee2cdb0a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:06:48.632679 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16b6e417-0ec4-4a40-853b-e969ccd1c578 None None] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:06:48.637319 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a05a7506-9424-4d9c-bfce-e5d6196d8b8e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:06:48.638185 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a05a7506-9424-4d9c-bfce-e5d6196d8b8e admin admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:06:48.638466 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a05a7506-9424-4d9c-bfce-e5d6196d8b8e admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:06:48.638739 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a05a7506-9424-4d9c-bfce-e5d6196d8b8e admin admin] PATH_INFO: `/v3/roles` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:06:48.640003 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a05a7506-9424-4d9c-bfce-e5d6196d8b8e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:06:48.641594 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a05a7506-9424-4d9c-bfce-e5d6196d8b8e admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:06:48.644354 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.auth.core [None req-42fecae6-b743-4dce-b0dd-f1edee2cdb0a None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81437) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:06:48.645546 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-42fecae6-b743-4dce-b0dd-f1edee2cdb0a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:06:48.651769 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 770/1560] 10.222.0.50 () {58 vars in 1122 bytes} [Tue Sep 13 16:06:48 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 16:06:48.653681 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65319ac7-237e-48a3-9d60-ce13da4ad077 None None] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:06:48.653968 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65319ac7-237e-48a3-9d60-ce13da4ad077 None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:06:48.654249 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65319ac7-237e-48a3-9d60-ce13da4ad077 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:06:48.656909 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 791/1561] 10.222.0.50 () {58 vars in 958 bytes} [Tue Sep 13 16:06:48 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 37 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:06:48.657760 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42fecae6-b743-4dce-b0dd-f1edee2cdb0a None None] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:06:48.667402 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42fecae6-b743-4dce-b0dd-f1edee2cdb0a None None] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:06:48.672023 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c30ad46-a2c3-4051-b11c-96eaf948a0ce admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:06:48.672611 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c30ad46-a2c3-4051-b11c-96eaf948a0ce admin admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:06:48.672886 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c30ad46-a2c3-4051-b11c-96eaf948a0ce admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:06:48.673154 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c30ad46-a2c3-4051-b11c-96eaf948a0ce admin admin] PATH_INFO: `/v3/roles` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:06:48.673782 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c30ad46-a2c3-4051-b11c-96eaf948a0ce admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:06:48.674316 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.auth.core [None req-65319ac7-237e-48a3-9d60-ce13da4ad077 None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81438) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:06:48.675405 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c30ad46-a2c3-4051-b11c-96eaf948a0ce admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:06:48.675550 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-65319ac7-237e-48a3-9d60-ce13da4ad077 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:06:48.676715 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:06:48.684418 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 771/1562] 10.222.0.50 () {58 vars in 958 bytes} [Tue Sep 13 16:06:48 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:06:48.685620 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5336958b-0691-4e38-919b-ae7044f982c8 None None] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:06:48.686484 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5336958b-0691-4e38-919b-ae7044f982c8 None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:06:48.686767 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5336958b-0691-4e38-919b-ae7044f982c8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:06:48.687472 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 792/1563] 10.222.0.50 () {58 vars in 1122 bytes} [Tue Sep 13 16:06:48 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 16:06:48.688882 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6b6c7b2-fe98-496b-a477-e2a9aeba2a23 None None] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:06:48.689178 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6b6c7b2-fe98-496b-a477-e2a9aeba2a23 None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:06:48.689660 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6b6c7b2-fe98-496b-a477-e2a9aeba2a23 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:06:48.709328 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.auth.core [None req-d6b6c7b2-fe98-496b-a477-e2a9aeba2a23 None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81437) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:06:48.709531 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.auth.core [None req-5336958b-0691-4e38-919b-ae7044f982c8 None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81438) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:06:48.710512 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-d6b6c7b2-fe98-496b-a477-e2a9aeba2a23 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:06:48.710628 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-5336958b-0691-4e38-919b-ae7044f982c8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:06:48.718144 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 793/1564] 10.222.0.50 () {58 vars in 958 bytes} [Tue Sep 13 16:06:48 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:06:48.722008 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 772/1565] 10.222.0.50 () {58 vars in 958 bytes} [Tue Sep 13 16:06:48 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:06:48.722175 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6b6c7b2-fe98-496b-a477-e2a9aeba2a23 None None] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:06:48.722277 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5336958b-0691-4e38-919b-ae7044f982c8 None None] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:06:48.731460 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6b6c7b2-fe98-496b-a477-e2a9aeba2a23 None None] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:06:48.731636 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5336958b-0691-4e38-919b-ae7044f982c8 None None] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:06:48.735958 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bc1ee6c-e7fa-4312-a773-58cf1a518162 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:06:48.736207 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5995de1a-0bc4-42c6-b260-9ff5afafd70f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:06:48.736540 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bc1ee6c-e7fa-4312-a773-58cf1a518162 admin admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:06:48.736796 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5995de1a-0bc4-42c6-b260-9ff5afafd70f admin admin] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:06:48.736904 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bc1ee6c-e7fa-4312-a773-58cf1a518162 admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:06:48.737068 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bc1ee6c-e7fa-4312-a773-58cf1a518162 admin admin] PATH_INFO: `/v3/domains` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:06:48.737173 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5995de1a-0bc4-42c6-b260-9ff5afafd70f admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:06:48.737339 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5995de1a-0bc4-42c6-b260-9ff5afafd70f admin admin] PATH_INFO: `/v3/projects` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:06:48.737700 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bc1ee6c-e7fa-4312-a773-58cf1a518162 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81437) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 16:06:48.737960 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5995de1a-0bc4-42c6-b260-9ff5afafd70f admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:06:48.738073 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bc1ee6c-e7fa-4312-a773-58cf1a518162 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:06:48.739604 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bc1ee6c-e7fa-4312-a773-58cf1a518162 admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:06:48.739884 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5995de1a-0bc4-42c6-b260-9ff5afafd70f admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:06:48.751646 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 794/1566] 10.222.0.50 () {58 vars in 1153 bytes} [Tue Sep 13 16:06:48 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 16:06:48.752511 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bc1ee6c-e7fa-4312-a773-58cf1a518162 admin admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:06:48.762155 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bc1ee6c-e7fa-4312-a773-58cf1a518162 admin admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:06:48.768007 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87a821e3-6c7e-4aa4-95d1-be875fadbf80 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:06:48.768605 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87a821e3-6c7e-4aa4-95d1-be875fadbf80 admin admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:06:48.772143 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87a821e3-6c7e-4aa4-95d1-be875fadbf80 admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:06:48.772361 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 773/1567] 10.222.0.50 () {60 vars in 1153 bytes} [Tue Sep 13 16:06:48 2022] POST /identity/v3/projects => generated 389 bytes in 50 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 16:06:48.772492 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87a821e3-6c7e-4aa4-95d1-be875fadbf80 admin admin] PATH_INFO: `/v3/roles` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:06:48.772598 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5995de1a-0bc4-42c6-b260-9ff5afafd70f admin admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:06:48.772727 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87a821e3-6c7e-4aa4-95d1-be875fadbf80 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:06:48.772727 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87a821e3-6c7e-4aa4-95d1-be875fadbf80 admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:06:48.786402 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 795/1568] 10.222.0.50 () {58 vars in 1122 bytes} [Tue Sep 13 16:06:48 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 16:06:48.787220 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87a821e3-6c7e-4aa4-95d1-be875fadbf80 admin admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:06:48.797208 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87a821e3-6c7e-4aa4-95d1-be875fadbf80 admin admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:06:48.832610 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5995de1a-0bc4-42c6-b260-9ff5afafd70f admin admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:06:48.837265 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fce47f8-38b0-449d-b06e-3cf1fa6e2046 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:06:48.837865 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fce47f8-38b0-449d-b06e-3cf1fa6e2046 admin admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:06:48.838145 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fce47f8-38b0-449d-b06e-3cf1fa6e2046 admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:06:48.838416 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fce47f8-38b0-449d-b06e-3cf1fa6e2046 admin admin] PATH_INFO: `/v3/domains` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:06:48.839092 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fce47f8-38b0-449d-b06e-3cf1fa6e2046 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81438) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 16:06:48.839463 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fce47f8-38b0-449d-b06e-3cf1fa6e2046 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:06:48.841059 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fce47f8-38b0-449d-b06e-3cf1fa6e2046 admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:06:48.853673 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 774/1569] 10.222.0.50 () {58 vars in 1153 bytes} [Tue Sep 13 16:06:48 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 84 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 16:06:48.854474 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fce47f8-38b0-449d-b06e-3cf1fa6e2046 admin admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:06:48.862030 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-122ceb43-14f7-4161-865e-86b5137eab9a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:06:48.862714 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-122ceb43-14f7-4161-865e-86b5137eab9a admin admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:06:48.863084 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-122ceb43-14f7-4161-865e-86b5137eab9a admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:06:48.863373 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-122ceb43-14f7-4161-865e-86b5137eab9a admin admin] PATH_INFO: `/v3/domains` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:06:48.864021 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-122ceb43-14f7-4161-865e-86b5137eab9a admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81437) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 16:06:48.864324 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-122ceb43-14f7-4161-865e-86b5137eab9a admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:06:48.864594 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fce47f8-38b0-449d-b06e-3cf1fa6e2046 admin admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:06:48.866195 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-122ceb43-14f7-4161-865e-86b5137eab9a admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:06:48.872615 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55aa4101-c20d-4da6-847c-f45203048a81 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:06:48.873881 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55aa4101-c20d-4da6-847c-f45203048a81 admin admin] REQUEST_METHOD: `PUT` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:06:48.874271 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55aa4101-c20d-4da6-847c-f45203048a81 admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:06:48.874606 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55aa4101-c20d-4da6-847c-f45203048a81 admin admin] PATH_INFO: `/v3/projects/8be8793418084985bce6350626212347/users/ff06e46c42034f59b393ce213830cfa5/roles/15db4aaca3264ae29bde38903ba43471` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:06:48.879997 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 796/1570] 10.222.0.50 () {58 vars in 1153 bytes} [Tue Sep 13 16:06:48 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 93 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 16:06:48.883929 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-122ceb43-14f7-4161-865e-86b5137eab9a admin admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:06:48.892830 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55aa4101-c20d-4da6-847c-f45203048a81 admin admin] RBAC: Authorizing `identity:create_grant(project_id=8be8793418084985bce6350626212347, user_id=ff06e46c42034f59b393ce213830cfa5, role_id=15db4aaca3264ae29bde38903ba43471)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:06:48.895656 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55aa4101-c20d-4da6-847c-f45203048a81 admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:06:48.895780 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-122ceb43-14f7-4161-865e-86b5137eab9a admin admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:06:48.900435 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67f3524e-2b0a-40a6-bc08-f9525cc1c149 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:06:48.901096 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67f3524e-2b0a-40a6-bc08-f9525cc1c149 admin admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:06:48.901418 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67f3524e-2b0a-40a6-bc08-f9525cc1c149 admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:06:48.901722 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67f3524e-2b0a-40a6-bc08-f9525cc1c149 admin admin] PATH_INFO: `/v3/domains` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:06:48.902371 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67f3524e-2b0a-40a6-bc08-f9525cc1c149 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81437) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 16:06:48.902681 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67f3524e-2b0a-40a6-bc08-f9525cc1c149 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:06:48.904363 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67f3524e-2b0a-40a6-bc08-f9525cc1c149 admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:06:48.917454 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 775/1571] 10.222.0.50 () {60 vars in 1483 bytes} [Tue Sep 13 16:06:48 2022] PUT /identity/v3/projects/8be8793418084985bce6350626212347/users/ff06e46c42034f59b393ce213830cfa5/roles/15db4aaca3264ae29bde38903ba43471 => generated 0 bytes in 55 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 16:06:48.917454 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55aa4101-c20d-4da6-847c-f45203048a81 admin admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:06:48.917750 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 797/1572] 10.222.0.50 () {58 vars in 1153 bytes} [Tue Sep 13 16:06:48 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 16:06:48.918055 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67f3524e-2b0a-40a6-bc08-f9525cc1c149 admin admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:06:48.928057 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67f3524e-2b0a-40a6-bc08-f9525cc1c149 admin admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:06:48.985331 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10052094-275d-43af-94bf-de216017c90f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:06:48.985602 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55aa4101-c20d-4da6-847c-f45203048a81 admin admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:06:48.985975 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10052094-275d-43af-94bf-de216017c90f admin admin] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:06:48.986257 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10052094-275d-43af-94bf-de216017c90f admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:06:48.986532 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10052094-275d-43af-94bf-de216017c90f admin admin] PATH_INFO: `/v3/projects` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:06:48.987238 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10052094-275d-43af-94bf-de216017c90f admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:06:48.988265 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: WARNING py.warnings [None req-10052094-275d-43af-94bf-de216017c90f admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_project": "(role:admin and system_scope:all) or (role:admin and domain_id:%(target.project.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Sep 13 16:06:48.988265 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: warnings.warn(msg) Sep 13 16:06:48.988265 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: Sep 13 16:06:48.989492 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10052094-275d-43af-94bf-de216017c90f admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:06:48.990332 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f7846cd-e116-43b4-b7b0-f579d23b437b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:06:48.991034 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f7846cd-e116-43b4-b7b0-f579d23b437b admin admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:06:48.991316 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f7846cd-e116-43b4-b7b0-f579d23b437b admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:06:48.991588 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f7846cd-e116-43b4-b7b0-f579d23b437b admin admin] PATH_INFO: `/v3/domains` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:06:48.992233 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f7846cd-e116-43b4-b7b0-f579d23b437b admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81438) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 16:06:48.992540 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f7846cd-e116-43b4-b7b0-f579d23b437b admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:06:48.994150 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f7846cd-e116-43b4-b7b0-f579d23b437b admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:06:49.006123 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 776/1573] 10.222.0.50 () {58 vars in 1153 bytes} [Tue Sep 13 16:06:48 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 96 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 16:06:49.006735 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f7846cd-e116-43b4-b7b0-f579d23b437b admin admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:06:49.018101 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f7846cd-e116-43b4-b7b0-f579d23b437b admin admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:06:49.021688 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 798/1574] 10.222.0.50 () {60 vars in 1153 bytes} [Tue Sep 13 16:06:48 2022] POST /identity/v3/projects => generated 385 bytes in 104 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 16:06:49.022487 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10052094-275d-43af-94bf-de216017c90f admin admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:06:49.025260 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8dbd5e42-75a9-46b0-8a87-0af26e433e96 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:06:49.025860 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dbd5e42-75a9-46b0-8a87-0af26e433e96 admin admin] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:06:49.026189 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dbd5e42-75a9-46b0-8a87-0af26e433e96 admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:06:49.026466 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dbd5e42-75a9-46b0-8a87-0af26e433e96 admin admin] PATH_INFO: `/v3/users` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:06:49.027113 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8dbd5e42-75a9-46b0-8a87-0af26e433e96 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:06:49.029013 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8dbd5e42-75a9-46b0-8a87-0af26e433e96 admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:06:49.049234 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:06:49.055668 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 777/1575] 10.222.0.50 () {60 vars in 1144 bytes} [Tue Sep 13 16:06:49 2022] POST /identity/v3/users => generated 295 bytes in 49 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 16:06:49.056564 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8dbd5e42-75a9-46b0-8a87-0af26e433e96 admin admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:06:49.062548 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:06:49.067694 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8dbd5e42-75a9-46b0-8a87-0af26e433e96 admin admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:06:49.096601 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10052094-275d-43af-94bf-de216017c90f admin admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:06:49.101312 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80cf643a-1193-477f-91a1-b7d644dff4f2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:06:49.102060 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80cf643a-1193-477f-91a1-b7d644dff4f2 admin admin] REQUEST_METHOD: `PUT` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:06:49.102332 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80cf643a-1193-477f-91a1-b7d644dff4f2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:06:49.102613 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80cf643a-1193-477f-91a1-b7d644dff4f2 admin admin] PATH_INFO: `/v3/projects/d57e69de18ac46399beb36eaca6bf06a/users/aa034994d95a4b82bf85138fdb07f4b4/roles/15db4aaca3264ae29bde38903ba43471` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:06:49.117818 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80cf643a-1193-477f-91a1-b7d644dff4f2 admin admin] RBAC: Authorizing `identity:create_grant(project_id=d57e69de18ac46399beb36eaca6bf06a, user_id=aa034994d95a4b82bf85138fdb07f4b4, role_id=15db4aaca3264ae29bde38903ba43471)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:06:49.118869 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: WARNING py.warnings [None req-80cf643a-1193-477f-91a1-b7d644dff4f2 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_grant": "(role:admin and system_scope:all) or ((role:admin and domain_id:%(target.user.domain_id)s and domain_id:%(target.project.domain_id)s) or (role:admin and domain_id:%(target.user.domain_id)s and domain_id:%(target.domain.id)s) or (role:admin and domain_id:%(target.group.domain_id)s and domain_id:%(target.project.domain_id)s) or (role:admin and domain_id:%(target.group.domain_id)s and domain_id:%(target.domain.id)s)) and (domain_id:%(target.role.domain_id)s or None:%(target.role.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Sep 13 16:06:49.118869 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: warnings.warn(msg) Sep 13 16:06:49.118869 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: Sep 13 16:06:49.120838 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80cf643a-1193-477f-91a1-b7d644dff4f2 admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:06:49.125641 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e682789-d3a0-4cbd-b340-6f6d7aec37ae admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:06:49.126316 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e682789-d3a0-4cbd-b340-6f6d7aec37ae admin admin] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:06:49.126626 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e682789-d3a0-4cbd-b340-6f6d7aec37ae admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:06:49.126967 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e682789-d3a0-4cbd-b340-6f6d7aec37ae admin admin] PATH_INFO: `/v3/projects` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:06:49.127706 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e682789-d3a0-4cbd-b340-6f6d7aec37ae admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:06:49.129668 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e682789-d3a0-4cbd-b340-6f6d7aec37ae admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:06:49.135827 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 799/1576] 10.222.0.50 () {60 vars in 1483 bytes} [Tue Sep 13 16:06:49 2022] PUT /identity/v3/projects/d57e69de18ac46399beb36eaca6bf06a/users/aa034994d95a4b82bf85138fdb07f4b4/roles/15db4aaca3264ae29bde38903ba43471 => generated 0 bytes in 114 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 16:06:49.140358 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80cf643a-1193-477f-91a1-b7d644dff4f2 admin admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:06:49.157380 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 778/1577] 10.222.0.50 () {60 vars in 1153 bytes} [Tue Sep 13 16:06:49 2022] POST /identity/v3/projects => generated 403 bytes in 101 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 16:06:49.158191 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e682789-d3a0-4cbd-b340-6f6d7aec37ae admin admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:06:49.205393 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80cf643a-1193-477f-91a1-b7d644dff4f2 admin admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:06:49.222188 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e682789-d3a0-4cbd-b340-6f6d7aec37ae admin admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:06:49.226928 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-193c9627-8571-48a9-8c06-a3080b89fd9e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:06:49.227604 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-193c9627-8571-48a9-8c06-a3080b89fd9e admin admin] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:06:49.227889 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-193c9627-8571-48a9-8c06-a3080b89fd9e admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:06:49.228165 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-193c9627-8571-48a9-8c06-a3080b89fd9e admin admin] PATH_INFO: `/v3/projects` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:06:49.228811 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-193c9627-8571-48a9-8c06-a3080b89fd9e admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:06:49.230714 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-193c9627-8571-48a9-8c06-a3080b89fd9e admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:06:49.258118 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 779/1578] 10.222.0.50 () {60 vars in 1153 bytes} [Tue Sep 13 16:06:49 2022] POST /identity/v3/projects => generated 379 bytes in 100 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 16:06:49.259650 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a8e45c1-71d2-4d02-9b6b-c0e5236477d0 None None] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:06:49.259938 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a8e45c1-71d2-4d02-9b6b-c0e5236477d0 None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:06:49.260212 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a8e45c1-71d2-4d02-9b6b-c0e5236477d0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:06:49.264601 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7442ddb3-6e69-49cc-a6de-83e142d89357 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:06:49.265220 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7442ddb3-6e69-49cc-a6de-83e142d89357 admin admin] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:06:49.265481 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7442ddb3-6e69-49cc-a6de-83e142d89357 admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:06:49.265862 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7442ddb3-6e69-49cc-a6de-83e142d89357 admin admin] PATH_INFO: `/v3/projects` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:06:49.266450 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7442ddb3-6e69-49cc-a6de-83e142d89357 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:06:49.268482 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7442ddb3-6e69-49cc-a6de-83e142d89357 admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:06:49.288907 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.auth.core [None req-2a8e45c1-71d2-4d02-9b6b-c0e5236477d0 None None] MFA Rules not processed for user `ff06e46c42034f59b393ce213830cfa5`. Rule list: `[]` (Enabled: `True`). {{(pid=81438) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:06:49.290196 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-2a8e45c1-71d2-4d02-9b6b-c0e5236477d0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:06:49.299528 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 800/1579] 10.222.0.50 () {60 vars in 1153 bytes} [Tue Sep 13 16:06:49 2022] POST /identity/v3/projects => generated 391 bytes in 160 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 16:06:49.299528 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7442ddb3-6e69-49cc-a6de-83e142d89357 admin admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:06:49.344732 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:06:49.360204 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:06:49.360818 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7442ddb3-6e69-49cc-a6de-83e142d89357 admin admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:06:49.365788 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfe9d304-b53f-4b20-b16f-99e57e0ba2a9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:06:49.366513 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfe9d304-b53f-4b20-b16f-99e57e0ba2a9 admin admin] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:06:49.366726 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 780/1580] 10.222.0.50 () {58 vars in 958 bytes} [Tue Sep 13 16:06:49 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:06:49.367060 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfe9d304-b53f-4b20-b16f-99e57e0ba2a9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:06:49.367377 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfe9d304-b53f-4b20-b16f-99e57e0ba2a9 admin admin] PATH_INFO: `/v3/projects` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:06:49.367907 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a8e45c1-71d2-4d02-9b6b-c0e5236477d0 None None] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:06:49.368044 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dfe9d304-b53f-4b20-b16f-99e57e0ba2a9 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:06:49.370194 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dfe9d304-b53f-4b20-b16f-99e57e0ba2a9 admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:06:49.410889 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 801/1581] 10.222.0.50 () {60 vars in 1153 bytes} [Tue Sep 13 16:06:49 2022] POST /identity/v3/projects => generated 377 bytes in 114 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 16:06:49.411757 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfe9d304-b53f-4b20-b16f-99e57e0ba2a9 admin admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:06:49.444246 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a8e45c1-71d2-4d02-9b6b-c0e5236477d0 None None] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:06:49.477541 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfe9d304-b53f-4b20-b16f-99e57e0ba2a9 admin admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:06:49.482213 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1961e51-cef4-4412-baf3-01beb7a42351 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:06:49.482853 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1961e51-cef4-4412-baf3-01beb7a42351 admin admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:06:49.483169 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1961e51-cef4-4412-baf3-01beb7a42351 admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:06:49.483446 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1961e51-cef4-4412-baf3-01beb7a42351 admin admin] PATH_INFO: `/v3/roles` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:06:49.484087 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1961e51-cef4-4412-baf3-01beb7a42351 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:06:49.485703 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1961e51-cef4-4412-baf3-01beb7a42351 admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:06:49.499123 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 802/1582] 10.222.0.50 () {58 vars in 1122 bytes} [Tue Sep 13 16:06:49 2022] GET /identity/v3/roles => generated 1420 bytes in 84 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 16:06:49.499123 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab80ffb0-dbd0-4244-9e9a-0a52019e677f None None] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:06:49.499123 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab80ffb0-dbd0-4244-9e9a-0a52019e677f None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:06:49.499123 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab80ffb0-dbd0-4244-9e9a-0a52019e677f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:06:49.500831 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f564f5c1-4109-4354-ae52-109f5d6ff371 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:06:49.501513 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f564f5c1-4109-4354-ae52-109f5d6ff371 admin admin] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:06:49.501870 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f564f5c1-4109-4354-ae52-109f5d6ff371 admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:06:49.502218 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f564f5c1-4109-4354-ae52-109f5d6ff371 admin admin] PATH_INFO: `/v3/users` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:06:49.502937 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f564f5c1-4109-4354-ae52-109f5d6ff371 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:06:49.504844 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f564f5c1-4109-4354-ae52-109f5d6ff371 admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:06:49.517402 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.auth.core [None req-ab80ffb0-dbd0-4244-9e9a-0a52019e677f None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81437) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:06:49.518722 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-ab80ffb0-dbd0-4244-9e9a-0a52019e677f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:06:49.526964 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 781/1583] 10.222.0.50 () {60 vars in 1144 bytes} [Tue Sep 13 16:06:49 2022] POST /identity/v3/users => generated 293 bytes in 160 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 16:06:49.527095 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 803/1584] 10.222.0.50 () {58 vars in 958 bytes} [Tue Sep 13 16:06:49 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:06:49.527685 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab80ffb0-dbd0-4244-9e9a-0a52019e677f None None] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:06:49.531764 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f564f5c1-4109-4354-ae52-109f5d6ff371 admin admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:06:49.541849 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f564f5c1-4109-4354-ae52-109f5d6ff371 admin admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:06:49.543735 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab80ffb0-dbd0-4244-9e9a-0a52019e677f None None] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:06:49.546793 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5aaae60a-31bc-443d-a99c-57f9a1bacf6b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:06:49.547512 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5aaae60a-31bc-443d-a99c-57f9a1bacf6b admin admin] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:06:49.547866 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5aaae60a-31bc-443d-a99c-57f9a1bacf6b admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:06:49.548209 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5aaae60a-31bc-443d-a99c-57f9a1bacf6b admin admin] PATH_INFO: `/v3/users` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:06:49.548403 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35a41ad8-41d9-434e-887b-343462f39f24 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:06:49.548993 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5aaae60a-31bc-443d-a99c-57f9a1bacf6b admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:06:49.549105 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35a41ad8-41d9-434e-887b-343462f39f24 admin admin] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:06:49.549273 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35a41ad8-41d9-434e-887b-343462f39f24 admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:06:49.549611 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35a41ad8-41d9-434e-887b-343462f39f24 admin admin] PATH_INFO: `/v3/users` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:06:49.550176 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35a41ad8-41d9-434e-887b-343462f39f24 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:06:49.551314 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5aaae60a-31bc-443d-a99c-57f9a1bacf6b admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:06:49.552188 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35a41ad8-41d9-434e-887b-343462f39f24 admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:06:49.573773 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 782/1585] 10.222.0.50 () {60 vars in 1144 bytes} [Tue Sep 13 16:06:49 2022] POST /identity/v3/users => generated 302 bytes in 46 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 16:06:49.574648 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5aaae60a-31bc-443d-a99c-57f9a1bacf6b admin admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:06:49.578334 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 804/1586] 10.222.0.50 () {60 vars in 1144 bytes} [Tue Sep 13 16:06:49 2022] POST /identity/v3/users => generated 290 bytes in 47 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 16:06:49.578334 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0ce918c-129a-44f2-9183-3ea816743a28 None None] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:06:49.578334 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0ce918c-129a-44f2-9183-3ea816743a28 None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:06:49.578334 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0ce918c-129a-44f2-9183-3ea816743a28 None None] PATH_INFO: `/` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:06:49.579145 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 805/1587] 10.222.0.50 () {54 vars in 962 bytes} [Tue Sep 13 16:06:49 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 16:06:49.579830 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0ce918c-129a-44f2-9183-3ea816743a28 None None] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:06:49.585737 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5aaae60a-31bc-443d-a99c-57f9a1bacf6b admin admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:06:49.590363 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cbb3916-1d60-47c6-a90b-366be7ab283e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:06:49.590989 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cbb3916-1d60-47c6-a90b-366be7ab283e admin admin] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:06:49.591288 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cbb3916-1d60-47c6-a90b-366be7ab283e admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:06:49.591559 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cbb3916-1d60-47c6-a90b-366be7ab283e admin admin] PATH_INFO: `/v3/users` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:06:49.592205 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2cbb3916-1d60-47c6-a90b-366be7ab283e admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:06:49.592608 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0ce918c-129a-44f2-9183-3ea816743a28 None None] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:06:49.594110 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2cbb3916-1d60-47c6-a90b-366be7ab283e admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:06:49.597125 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a68835b-e89c-4e6c-bda1-77b58351d724 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:06:49.597726 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a68835b-e89c-4e6c-bda1-77b58351d724 admin admin] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:06:49.598006 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a68835b-e89c-4e6c-bda1-77b58351d724 admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:06:49.598280 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a68835b-e89c-4e6c-bda1-77b58351d724 admin admin] PATH_INFO: `/v3/users` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:06:49.598910 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a68835b-e89c-4e6c-bda1-77b58351d724 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:06:49.600825 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a68835b-e89c-4e6c-bda1-77b58351d724 admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:06:49.620508 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 783/1588] 10.222.0.50 () {60 vars in 1144 bytes} [Tue Sep 13 16:06:49 2022] POST /identity/v3/users => generated 296 bytes in 43 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 16:06:49.620508 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cbb3916-1d60-47c6-a90b-366be7ab283e admin admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:06:49.625191 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 806/1589] 10.222.0.50 () {60 vars in 1144 bytes} [Tue Sep 13 16:06:49 2022] POST /identity/v3/users => generated 289 bytes in 42 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 16:06:49.625191 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a68835b-e89c-4e6c-bda1-77b58351d724 admin admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:06:49.630692 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cbb3916-1d60-47c6-a90b-366be7ab283e admin admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:06:49.633162 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a68835b-e89c-4e6c-bda1-77b58351d724 admin admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:06:49.635234 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb7b9f97-6413-4f99-8bf0-80b3123796b0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:06:49.635822 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb7b9f97-6413-4f99-8bf0-80b3123796b0 admin admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:06:49.636103 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb7b9f97-6413-4f99-8bf0-80b3123796b0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:06:49.636371 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb7b9f97-6413-4f99-8bf0-80b3123796b0 admin admin] PATH_INFO: `/v3/roles` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:06:49.637112 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb7b9f97-6413-4f99-8bf0-80b3123796b0 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:06:49.637982 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba5fd967-87ac-485e-b6aa-1ec09733d286 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:06:49.638561 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba5fd967-87ac-485e-b6aa-1ec09733d286 admin admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:06:49.638887 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba5fd967-87ac-485e-b6aa-1ec09733d286 admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:06:49.639714 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb7b9f97-6413-4f99-8bf0-80b3123796b0 admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:06:49.639850 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba5fd967-87ac-485e-b6aa-1ec09733d286 admin admin] PATH_INFO: `/v3/roles` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:06:49.640349 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba5fd967-87ac-485e-b6aa-1ec09733d286 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:06:49.642506 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba5fd967-87ac-485e-b6aa-1ec09733d286 admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:06:49.649759 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 784/1590] 10.222.0.50 () {58 vars in 1122 bytes} [Tue Sep 13 16:06:49 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 16:06:49.650386 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb7b9f97-6413-4f99-8bf0-80b3123796b0 admin admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:06:49.652034 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-bb7b9f97-6413-4f99-8bf0-80b3123796b0 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:06:49.656242 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 807/1591] 10.222.0.50 () {58 vars in 1122 bytes} [Tue Sep 13 16:06:49 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 16:06:49.657059 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba5fd967-87ac-485e-b6aa-1ec09733d286 admin admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:06:49.666417 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba5fd967-87ac-485e-b6aa-1ec09733d286 admin admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:06:49.671596 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cc69f29-6b4a-4330-b12a-fd66d395ad78 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:06:49.672181 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cc69f29-6b4a-4330-b12a-fd66d395ad78 admin admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:06:49.672457 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cc69f29-6b4a-4330-b12a-fd66d395ad78 admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:06:49.672808 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cc69f29-6b4a-4330-b12a-fd66d395ad78 admin admin] PATH_INFO: `/v3/roles` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:06:49.673486 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0cc69f29-6b4a-4330-b12a-fd66d395ad78 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:06:49.675207 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0cc69f29-6b4a-4330-b12a-fd66d395ad78 admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:06:49.688724 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 808/1592] 10.222.0.50 () {58 vars in 1122 bytes} [Tue Sep 13 16:06:49 2022] GET /identity/v3/roles => generated 1420 bytes in 28 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 16:06:49.688724 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cc69f29-6b4a-4330-b12a-fd66d395ad78 admin admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:06:49.695879 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cc69f29-6b4a-4330-b12a-fd66d395ad78 admin admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:06:49.700780 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56138c47-d83d-4a16-83f8-5e2f97db1c4d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:06:49.701438 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56138c47-d83d-4a16-83f8-5e2f97db1c4d admin admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:06:49.701823 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56138c47-d83d-4a16-83f8-5e2f97db1c4d admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:06:49.702230 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56138c47-d83d-4a16-83f8-5e2f97db1c4d admin admin] PATH_INFO: `/v3/roles` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:06:49.702791 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56138c47-d83d-4a16-83f8-5e2f97db1c4d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:06:49.704451 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56138c47-d83d-4a16-83f8-5e2f97db1c4d admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:06:49.709963 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=91,keystone:INSERT=9 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:06:49.717679 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 809/1593] 10.222.0.50 () {58 vars in 1122 bytes} [Tue Sep 13 16:06:49 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 16:06:49.717942 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb7b9f97-6413-4f99-8bf0-80b3123796b0 admin admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:06:49.718058 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb131762-92e1-4d46-9ded-bc84b92e27a0 None None] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:06:49.718058 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb131762-92e1-4d46-9ded-bc84b92e27a0 None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:06:49.718058 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb131762-92e1-4d46-9ded-bc84b92e27a0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:06:49.718769 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcdbf838-5057-4cc7-9585-41d5cdfd7731 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f436f07939d4855a21b195bc7566244', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:06:49.719727 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcdbf838-5057-4cc7-9585-41d5cdfd7731 service neutron] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:06:49.719978 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcdbf838-5057-4cc7-9585-41d5cdfd7731 service neutron] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:06:49.720449 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcdbf838-5057-4cc7-9585-41d5cdfd7731 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:06:49.730115 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bcdbf838-5057-4cc7-9585-41d5cdfd7731 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:06:49.731587 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bcdbf838-5057-4cc7-9585-41d5cdfd7731 service neutron] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:06:49.741746 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 785/1594] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 16:06:49 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 92 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 16:06:49.742569 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcdbf838-5057-4cc7-9585-41d5cdfd7731 service neutron] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:06:49.753239 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcdbf838-5057-4cc7-9585-41d5cdfd7731 service neutron] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:06:49.757805 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.auth.core [None req-eb131762-92e1-4d46-9ded-bc84b92e27a0 None None] MFA Rules not processed for user `4e039b46c5e3445bbb39904b00c5430e`. Rule list: `[]` (Enabled: `True`). {{(pid=81437) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:06:49.758965 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fff3c515-346d-4b02-a3d8-edfb0c5b9b24 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:06:49.759109 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-eb131762-92e1-4d46-9ded-bc84b92e27a0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:06:49.759875 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fff3c515-346d-4b02-a3d8-edfb0c5b9b24 admin admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:06:49.760351 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fff3c515-346d-4b02-a3d8-edfb0c5b9b24 admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:06:49.760795 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fff3c515-346d-4b02-a3d8-edfb0c5b9b24 admin admin] PATH_INFO: `/v3/roles` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:06:49.761580 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fff3c515-346d-4b02-a3d8-edfb0c5b9b24 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:06:49.763474 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fff3c515-346d-4b02-a3d8-edfb0c5b9b24 admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:06:49.775501 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 786/1595] 10.222.0.50 () {58 vars in 1122 bytes} [Tue Sep 13 16:06:49 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 16:06:49.776200 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fff3c515-346d-4b02-a3d8-edfb0c5b9b24 admin admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:06:49.786667 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fff3c515-346d-4b02-a3d8-edfb0c5b9b24 admin admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:06:49.792067 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3584667f-7c6c-496b-9e55-c0d20d211b76 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:06:49.792670 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3584667f-7c6c-496b-9e55-c0d20d211b76 admin admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:06:49.793017 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3584667f-7c6c-496b-9e55-c0d20d211b76 admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:06:49.793335 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3584667f-7c6c-496b-9e55-c0d20d211b76 admin admin] PATH_INFO: `/v3/roles` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:06:49.793842 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3584667f-7c6c-496b-9e55-c0d20d211b76 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:06:49.795752 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3584667f-7c6c-496b-9e55-c0d20d211b76 admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:06:49.801345 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=93,keystone:INSERT=7 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:06:49.807559 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 787/1596] 10.222.0.50 () {58 vars in 1122 bytes} [Tue Sep 13 16:06:49 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 16:06:49.808587 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3584667f-7c6c-496b-9e55-c0d20d211b76 admin admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:06:49.821241 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3584667f-7c6c-496b-9e55-c0d20d211b76 admin admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:06:49.827683 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60161e61-0379-4cb5-b4f0-584e2950f6c2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:06:49.828556 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60161e61-0379-4cb5-b4f0-584e2950f6c2 admin admin] REQUEST_METHOD: `PUT` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:06:49.828839 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60161e61-0379-4cb5-b4f0-584e2950f6c2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:06:49.829112 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60161e61-0379-4cb5-b4f0-584e2950f6c2 admin admin] PATH_INFO: `/v3/projects/859cfa45a18845f49e525ab4ed54b529/users/c0fd4bbe430b4a8888f3be4b23982c5a/roles/15db4aaca3264ae29bde38903ba43471` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:06:49.844493 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 810/1597] 10.222.0.50 () {58 vars in 1059 bytes} [Tue Sep 13 16:06:49 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 129 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:06:49.845359 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb131762-92e1-4d46-9ded-bc84b92e27a0 None None] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:06:49.847793 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60161e61-0379-4cb5-b4f0-584e2950f6c2 admin admin] RBAC: Authorizing `identity:create_grant(project_id=859cfa45a18845f49e525ab4ed54b529, user_id=c0fd4bbe430b4a8888f3be4b23982c5a, role_id=15db4aaca3264ae29bde38903ba43471)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:06:49.850824 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60161e61-0379-4cb5-b4f0-584e2950f6c2 admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:06:49.858381 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb131762-92e1-4d46-9ded-bc84b92e27a0 None None] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:06:49.869137 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 788/1598] 10.222.0.50 () {60 vars in 1483 bytes} [Tue Sep 13 16:06:49 2022] PUT /identity/v3/projects/859cfa45a18845f49e525ab4ed54b529/users/c0fd4bbe430b4a8888f3be4b23982c5a/roles/15db4aaca3264ae29bde38903ba43471 => generated 0 bytes in 61 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 16:06:49.871246 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60161e61-0379-4cb5-b4f0-584e2950f6c2 admin admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:06:49.937049 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-328efa11-3616-4539-ad05-844f080ade9b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:06:49.937647 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-328efa11-3616-4539-ad05-844f080ade9b admin admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:06:49.937938 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-328efa11-3616-4539-ad05-844f080ade9b admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:06:49.938270 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-328efa11-3616-4539-ad05-844f080ade9b admin admin] PATH_INFO: `/v3/roles` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:06:49.938850 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-328efa11-3616-4539-ad05-844f080ade9b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:06:49.940504 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-328efa11-3616-4539-ad05-844f080ade9b admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:06:49.946301 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60161e61-0379-4cb5-b4f0-584e2950f6c2 admin admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:06:49.951235 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-237f517d-dfd1-4e59-b56e-694860f4aa3d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:06:49.952447 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-237f517d-dfd1-4e59-b56e-694860f4aa3d admin admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:06:49.952934 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 811/1599] 10.222.0.50 () {58 vars in 1122 bytes} [Tue Sep 13 16:06:49 2022] GET /identity/v3/roles => generated 1420 bytes in 108 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 16:06:49.954256 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-328efa11-3616-4539-ad05-844f080ade9b admin admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:06:49.955974 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-237f517d-dfd1-4e59-b56e-694860f4aa3d admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:06:49.956261 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-237f517d-dfd1-4e59-b56e-694860f4aa3d admin admin] PATH_INFO: `/v3/roles` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:06:49.956929 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-237f517d-dfd1-4e59-b56e-694860f4aa3d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:06:49.958708 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-237f517d-dfd1-4e59-b56e-694860f4aa3d admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:06:49.968771 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-328efa11-3616-4539-ad05-844f080ade9b admin admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:06:49.973747 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d6df98c-53c4-437a-bba6-9967d81ad804 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:06:49.974461 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 789/1600] 10.222.0.50 () {58 vars in 1122 bytes} [Tue Sep 13 16:06:49 2022] GET /identity/v3/roles => generated 1420 bytes in 105 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 16:06:49.974937 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d6df98c-53c4-437a-bba6-9967d81ad804 admin admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:06:49.975247 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-237f517d-dfd1-4e59-b56e-694860f4aa3d admin admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:06:49.975384 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d6df98c-53c4-437a-bba6-9967d81ad804 admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:06:49.975557 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d6df98c-53c4-437a-bba6-9967d81ad804 admin admin] PATH_INFO: `/v3/roles` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:06:49.976312 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d6df98c-53c4-437a-bba6-9967d81ad804 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:06:49.978045 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d6df98c-53c4-437a-bba6-9967d81ad804 admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:06:49.986798 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-237f517d-dfd1-4e59-b56e-694860f4aa3d admin admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:06:49.993408 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 812/1601] 10.222.0.50 () {58 vars in 1122 bytes} [Tue Sep 13 16:06:49 2022] GET /identity/v3/roles => generated 1420 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 16:06:49.993408 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d6df98c-53c4-437a-bba6-9967d81ad804 admin admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:06:49.995066 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-843b0c0b-a40c-4f03-afc0-9d8ee48cb624 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:06:49.995684 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-843b0c0b-a40c-4f03-afc0-9d8ee48cb624 admin admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:06:49.995968 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-843b0c0b-a40c-4f03-afc0-9d8ee48cb624 admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:06:49.996243 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-843b0c0b-a40c-4f03-afc0-9d8ee48cb624 admin admin] PATH_INFO: `/v3/roles` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:06:49.996884 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-843b0c0b-a40c-4f03-afc0-9d8ee48cb624 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:06:49.999160 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-843b0c0b-a40c-4f03-afc0-9d8ee48cb624 admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:06:50.003995 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d6df98c-53c4-437a-bba6-9967d81ad804 admin admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:06:50.008713 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3978b6bb-8e51-49f8-b17d-07ab999e3f33 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:06:50.009317 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3978b6bb-8e51-49f8-b17d-07ab999e3f33 admin admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:06:50.009599 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3978b6bb-8e51-49f8-b17d-07ab999e3f33 admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:06:50.010009 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 790/1602] 10.222.0.50 () {58 vars in 1122 bytes} [Tue Sep 13 16:06:49 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 16:06:50.010736 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-843b0c0b-a40c-4f03-afc0-9d8ee48cb624 admin admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:06:50.013516 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3978b6bb-8e51-49f8-b17d-07ab999e3f33 admin admin] PATH_INFO: `/v3/roles` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:06:50.014460 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3978b6bb-8e51-49f8-b17d-07ab999e3f33 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:06:50.016652 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3978b6bb-8e51-49f8-b17d-07ab999e3f33 admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:06:50.031964 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 813/1603] 10.222.0.50 () {58 vars in 1122 bytes} [Tue Sep 13 16:06:49 2022] GET /identity/v3/roles => generated 1420 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 16:06:50.031964 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b92a0df9-77bc-44a5-95cf-7a76502b3c54 None None] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:06:50.031964 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b92a0df9-77bc-44a5-95cf-7a76502b3c54 None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:06:50.031964 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b92a0df9-77bc-44a5-95cf-7a76502b3c54 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:06:50.062473 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.auth.core [None req-b92a0df9-77bc-44a5-95cf-7a76502b3c54 None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81437) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:06:50.062473 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-b92a0df9-77bc-44a5-95cf-7a76502b3c54 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:06:50.079086 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 814/1604] 10.222.0.50 () {58 vars in 958 bytes} [Tue Sep 13 16:06:50 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 50 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:06:50.080570 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b92a0df9-77bc-44a5-95cf-7a76502b3c54 None None] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:06:50.094813 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b92a0df9-77bc-44a5-95cf-7a76502b3c54 None None] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:06:50.098244 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-843b0c0b-a40c-4f03-afc0-9d8ee48cb624 admin admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:06:50.099734 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3df30ec7-60a1-488f-94ef-3a32c2c35400 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:06:50.100399 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3df30ec7-60a1-488f-94ef-3a32c2c35400 admin admin] REQUEST_METHOD: `PUT` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:06:50.100669 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3df30ec7-60a1-488f-94ef-3a32c2c35400 admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:06:50.100941 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3df30ec7-60a1-488f-94ef-3a32c2c35400 admin admin] PATH_INFO: `/v3/projects/e907c12bca4b4205a4877e3c6596db16/users/5d421769b0c84034b83f6336f71d72bf/roles/15db4aaca3264ae29bde38903ba43471` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:06:50.104773 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa460800-7c76-4a42-a666-c31c634f4f30 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4e039b46c5e3445bbb39904b00c5430e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'admin', 'ResellerAdmin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:06:50.106069 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa460800-7c76-4a42-a666-c31c634f4f30 service nova] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:06:50.106354 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa460800-7c76-4a42-a666-c31c634f4f30 service nova] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:06:50.106671 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa460800-7c76-4a42-a666-c31c634f4f30 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:06:50.118178 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3df30ec7-60a1-488f-94ef-3a32c2c35400 admin admin] RBAC: Authorizing `identity:create_grant(project_id=e907c12bca4b4205a4877e3c6596db16, user_id=5d421769b0c84034b83f6336f71d72bf, role_id=15db4aaca3264ae29bde38903ba43471)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:06:50.120892 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3df30ec7-60a1-488f-94ef-3a32c2c35400 admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:06:50.144488 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 815/1605] 10.222.0.50 () {60 vars in 1483 bytes} [Tue Sep 13 16:06:50 2022] PUT /identity/v3/projects/e907c12bca4b4205a4877e3c6596db16/users/5d421769b0c84034b83f6336f71d72bf/roles/15db4aaca3264ae29bde38903ba43471 => generated 0 bytes in 56 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 16:06:50.144488 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3df30ec7-60a1-488f-94ef-3a32c2c35400 admin admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:06:50.147267 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:06:50.191806 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa460800-7c76-4a42-a666-c31c634f4f30 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:06:50.193328 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa460800-7c76-4a42-a666-c31c634f4f30 service nova] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:06:50.226218 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 791/1606] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 16:06:50 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 215 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 16:06:50.228925 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa460800-7c76-4a42-a666-c31c634f4f30 service nova] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:06:50.245333 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3df30ec7-60a1-488f-94ef-3a32c2c35400 admin admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:06:50.250769 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7ed8b0c-8a72-4232-9cab-9cec09713a7c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:06:50.251697 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7ed8b0c-8a72-4232-9cab-9cec09713a7c admin admin] REQUEST_METHOD: `PUT` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:06:50.252089 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7ed8b0c-8a72-4232-9cab-9cec09713a7c admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:06:50.252471 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7ed8b0c-8a72-4232-9cab-9cec09713a7c admin admin] PATH_INFO: `/v3/projects/3a13f08db2354b399d2be343bb888d3f/users/6f358deec606446a9fc085dcd4817d08/roles/15db4aaca3264ae29bde38903ba43471` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:06:50.254871 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa460800-7c76-4a42-a666-c31c634f4f30 service nova] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:06:50.270563 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42a68542-8baa-44b6-a7e4-29c611984582 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:06:50.271535 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42a68542-8baa-44b6-a7e4-29c611984582 admin admin] REQUEST_METHOD: `PUT` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:06:50.271991 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42a68542-8baa-44b6-a7e4-29c611984582 admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:06:50.272635 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7ed8b0c-8a72-4232-9cab-9cec09713a7c admin admin] RBAC: Authorizing `identity:create_grant(project_id=3a13f08db2354b399d2be343bb888d3f, user_id=6f358deec606446a9fc085dcd4817d08, role_id=15db4aaca3264ae29bde38903ba43471)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:06:50.273088 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42a68542-8baa-44b6-a7e4-29c611984582 admin admin] PATH_INFO: `/v3/projects/a345ae605d56441e86b2dd45565655c5/users/179e2ae27b924db0a49b64502548525a/roles/15db4aaca3264ae29bde38903ba43471` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:06:50.275981 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7ed8b0c-8a72-4232-9cab-9cec09713a7c admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:06:50.291762 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:06:50.299694 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 816/1607] 10.222.0.50 () {60 vars in 1483 bytes} [Tue Sep 13 16:06:50 2022] PUT /identity/v3/projects/3a13f08db2354b399d2be343bb888d3f/users/6f358deec606446a9fc085dcd4817d08/roles/15db4aaca3264ae29bde38903ba43471 => generated 0 bytes in 163 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 16:06:50.313151 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7ed8b0c-8a72-4232-9cab-9cec09713a7c admin admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:06:50.316222 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42a68542-8baa-44b6-a7e4-29c611984582 admin admin] RBAC: Authorizing `identity:create_grant(project_id=a345ae605d56441e86b2dd45565655c5, user_id=179e2ae27b924db0a49b64502548525a, role_id=15db4aaca3264ae29bde38903ba43471)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:06:50.319858 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42a68542-8baa-44b6-a7e4-29c611984582 admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:06:50.338509 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 792/1608] 10.222.0.50 () {60 vars in 1483 bytes} [Tue Sep 13 16:06:50 2022] PUT /identity/v3/projects/a345ae605d56441e86b2dd45565655c5/users/179e2ae27b924db0a49b64502548525a/roles/15db4aaca3264ae29bde38903ba43471 => generated 0 bytes in 110 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 16:06:50.339606 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42a68542-8baa-44b6-a7e4-29c611984582 admin admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:06:50.403724 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7ed8b0c-8a72-4232-9cab-9cec09713a7c admin admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:06:50.414856 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42a68542-8baa-44b6-a7e4-29c611984582 admin admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:06:50.425740 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d0bbecc-6e7a-4ee9-80b9-6c2536e8e8c4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:06:50.426548 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d0bbecc-6e7a-4ee9-80b9-6c2536e8e8c4 admin admin] REQUEST_METHOD: `PUT` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:06:50.427106 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d0bbecc-6e7a-4ee9-80b9-6c2536e8e8c4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:06:50.427452 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d0bbecc-6e7a-4ee9-80b9-6c2536e8e8c4 admin admin] PATH_INFO: `/v3/projects/4eb7191b18cc41de8c9c51672246991d/users/e1cbafa1d3c244c5906e475494b39d4c/roles/15db4aaca3264ae29bde38903ba43471` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:06:50.447735 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d0bbecc-6e7a-4ee9-80b9-6c2536e8e8c4 admin admin] RBAC: Authorizing `identity:create_grant(project_id=4eb7191b18cc41de8c9c51672246991d, user_id=e1cbafa1d3c244c5906e475494b39d4c, role_id=15db4aaca3264ae29bde38903ba43471)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:06:50.451229 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d0bbecc-6e7a-4ee9-80b9-6c2536e8e8c4 admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:06:50.469760 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 793/1609] 10.222.0.50 () {60 vars in 1483 bytes} [Tue Sep 13 16:06:50 2022] PUT /identity/v3/projects/4eb7191b18cc41de8c9c51672246991d/users/e1cbafa1d3c244c5906e475494b39d4c/roles/15db4aaca3264ae29bde38903ba43471 => generated 0 bytes in 131 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 16:06:50.473642 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d0bbecc-6e7a-4ee9-80b9-6c2536e8e8c4 admin admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:06:50.488454 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a21d3e7c-4fc3-40e7-9792-e55b47d7901a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:06:50.489455 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a21d3e7c-4fc3-40e7-9792-e55b47d7901a admin admin] REQUEST_METHOD: `PUT` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:06:50.489962 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a21d3e7c-4fc3-40e7-9792-e55b47d7901a admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:06:50.490601 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a21d3e7c-4fc3-40e7-9792-e55b47d7901a admin admin] PATH_INFO: `/v3/projects/5b11b198b8754bcabf244e0047475b8d/users/6d31b2ec391441f7bab3546d2354b0a1/roles/15db4aaca3264ae29bde38903ba43471` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:06:50.507837 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a21d3e7c-4fc3-40e7-9792-e55b47d7901a admin admin] RBAC: Authorizing `identity:create_grant(project_id=5b11b198b8754bcabf244e0047475b8d, user_id=6d31b2ec391441f7bab3546d2354b0a1, role_id=15db4aaca3264ae29bde38903ba43471)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:06:50.511030 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a21d3e7c-4fc3-40e7-9792-e55b47d7901a admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:06:50.537305 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 817/1610] 10.222.0.50 () {60 vars in 1483 bytes} [Tue Sep 13 16:06:50 2022] PUT /identity/v3/projects/5b11b198b8754bcabf244e0047475b8d/users/6d31b2ec391441f7bab3546d2354b0a1/roles/15db4aaca3264ae29bde38903ba43471 => generated 0 bytes in 237 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 16:06:50.538702 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6c92f79-3597-46a3-80b5-764610b9c7d2 None None] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:06:50.539263 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6c92f79-3597-46a3-80b5-764610b9c7d2 None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:06:50.539603 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6c92f79-3597-46a3-80b5-764610b9c7d2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:06:50.552285 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d0bbecc-6e7a-4ee9-80b9-6c2536e8e8c4 admin admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:06:50.578594 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.auth.core [None req-b6c92f79-3597-46a3-80b5-764610b9c7d2 None None] MFA Rules not processed for user `5d421769b0c84034b83f6336f71d72bf`. Rule list: `[]` (Enabled: `True`). {{(pid=81437) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:06:50.579975 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-b6c92f79-3597-46a3-80b5-764610b9c7d2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:06:50.602680 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:06:50.615352 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35170d29-a24d-4ade-86c5-5c6834301c2b service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f436f07939d4855a21b195bc7566244', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:06:50.616049 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35170d29-a24d-4ade-86c5-5c6834301c2b service neutron] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:06:50.616330 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35170d29-a24d-4ade-86c5-5c6834301c2b service neutron] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:06:50.616817 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35170d29-a24d-4ade-86c5-5c6834301c2b service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:06:50.656934 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 818/1611] 10.222.0.50 () {58 vars in 958 bytes} [Tue Sep 13 16:06:50 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:06:50.656934 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-268b6f13-019d-467b-b63b-440c0ece0bc5 None None] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:06:50.656934 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-268b6f13-019d-467b-b63b-440c0ece0bc5 None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:06:50.656934 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-268b6f13-019d-467b-b63b-440c0ece0bc5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:06:50.690297 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.auth.core [None req-268b6f13-019d-467b-b63b-440c0ece0bc5 None None] MFA Rules not processed for user `6f358deec606446a9fc085dcd4817d08`. Rule list: `[]` (Enabled: `True`). {{(pid=81437) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:06:50.691551 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-268b6f13-019d-467b-b63b-440c0ece0bc5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:06:50.695971 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35170d29-a24d-4ade-86c5-5c6834301c2b service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:06:50.697632 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35170d29-a24d-4ade-86c5-5c6834301c2b service neutron] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:06:50.697926 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:06:50.710165 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 794/1612] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 16:06:50 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 238 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 16:06:50.717725 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-054464c2-c9b9-4239-b31e-f2058b74df99 None None] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:06:50.718009 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-054464c2-c9b9-4239-b31e-f2058b74df99 None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:06:50.718374 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-054464c2-c9b9-4239-b31e-f2058b74df99 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:06:50.748300 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.auth.core [None req-054464c2-c9b9-4239-b31e-f2058b74df99 None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81438) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:06:50.750255 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-054464c2-c9b9-4239-b31e-f2058b74df99 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:06:50.760734 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 795/1613] 10.222.0.50 () {58 vars in 958 bytes} [Tue Sep 13 16:06:50 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 45 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:06:50.762193 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2699c507-b9b6-45e8-bab7-55adb5771084 None None] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:06:50.762743 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2699c507-b9b6-45e8-bab7-55adb5771084 None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:06:50.763324 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2699c507-b9b6-45e8-bab7-55adb5771084 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:06:50.787454 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.auth.core [None req-2699c507-b9b6-45e8-bab7-55adb5771084 None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81438) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:06:50.792168 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-2699c507-b9b6-45e8-bab7-55adb5771084 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:06:50.792318 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 819/1614] 10.222.0.50 () {58 vars in 958 bytes} [Tue Sep 13 16:06:50 2022] POST /identity/v3/auth/tokens => generated 3280 bytes in 136 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:06:50.792318 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c685e183-74bc-4c0b-8be5-8ffc892f252e None None] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:06:50.792318 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c685e183-74bc-4c0b-8be5-8ffc892f252e None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:06:50.792318 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c685e183-74bc-4c0b-8be5-8ffc892f252e None None] PATH_INFO: `/` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:06:50.792653 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 820/1615] 10.222.0.50 () {54 vars in 912 bytes} [Tue Sep 13 16:06:50 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 16:06:50.793731 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0513a65b-64d7-4928-b4f8-fb4d8a7e96e6 None None] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:06:50.794022 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0513a65b-64d7-4928-b4f8-fb4d8a7e96e6 None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:06:50.794313 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0513a65b-64d7-4928-b4f8-fb4d8a7e96e6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:06:50.807998 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 796/1616] 10.222.0.50 () {58 vars in 958 bytes} [Tue Sep 13 16:06:50 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 44 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:06:50.807998 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97ac6768-3575-4663-b53c-70439b8fb2e5 None None] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:06:50.807998 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97ac6768-3575-4663-b53c-70439b8fb2e5 None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:06:50.807998 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97ac6768-3575-4663-b53c-70439b8fb2e5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:06:50.815242 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.auth.core [None req-0513a65b-64d7-4928-b4f8-fb4d8a7e96e6 None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81437) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:06:50.816515 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-0513a65b-64d7-4928-b4f8-fb4d8a7e96e6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:06:50.829800 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 821/1617] 10.222.0.50 () {58 vars in 958 bytes} [Tue Sep 13 16:06:50 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:06:50.829800 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b702a2a1-ea94-4347-9821-f5b31d074034 None None] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:06:50.829800 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b702a2a1-ea94-4347-9821-f5b31d074034 None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:06:50.829800 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b702a2a1-ea94-4347-9821-f5b31d074034 None None] PATH_INFO: `/` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:06:50.830250 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 822/1618] 10.222.0.50 () {54 vars in 962 bytes} [Tue Sep 13 16:06:50 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 16:06:50.830849 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b702a2a1-ea94-4347-9821-f5b31d074034 None None] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:06:50.842714 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b702a2a1-ea94-4347-9821-f5b31d074034 None None] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:06:50.843081 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.auth.core [None req-97ac6768-3575-4663-b53c-70439b8fb2e5 None None] MFA Rules not processed for user `aa034994d95a4b82bf85138fdb07f4b4`. Rule list: `[]` (Enabled: `True`). {{(pid=81438) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:06:50.844477 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-97ac6768-3575-4663-b53c-70439b8fb2e5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:06:50.846788 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49575561-7603-4c69-9d75-d63e3ac0482d service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f436f07939d4855a21b195bc7566244', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:06:50.847480 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49575561-7603-4c69-9d75-d63e3ac0482d service neutron] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:06:50.847843 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49575561-7603-4c69-9d75-d63e3ac0482d service neutron] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:06:50.848106 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49575561-7603-4c69-9d75-d63e3ac0482d service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:06:50.859181 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49575561-7603-4c69-9d75-d63e3ac0482d service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:06:50.860710 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49575561-7603-4c69-9d75-d63e3ac0482d service neutron] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:06:50.874714 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 823/1619] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 16:06:50 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 44 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 16:06:50.877143 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49575561-7603-4c69-9d75-d63e3ac0482d service neutron] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:06:50.891343 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49575561-7603-4c69-9d75-d63e3ac0482d service neutron] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:06:50.897882 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18450311-b436-415f-b917-f4d35c165e76 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:06:50.898394 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18450311-b436-415f-b917-f4d35c165e76 admin admin] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:06:50.899079 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18450311-b436-415f-b917-f4d35c165e76 admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:06:50.899385 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18450311-b436-415f-b917-f4d35c165e76 admin admin] PATH_INFO: `/v3/projects` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:06:50.900714 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18450311-b436-415f-b917-f4d35c165e76 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:06:50.902530 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18450311-b436-415f-b917-f4d35c165e76 admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:06:50.935034 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 797/1620] 10.222.0.50 () {58 vars in 958 bytes} [Tue Sep 13 16:06:50 2022] POST /identity/v3/auth/tokens => generated 3282 bytes in 127 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:06:50.935034 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27d67d1e-2b06-48ce-9194-3332d9b9cef5 None None] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:06:50.935034 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27d67d1e-2b06-48ce-9194-3332d9b9cef5 None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:06:50.935034 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27d67d1e-2b06-48ce-9194-3332d9b9cef5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:06:50.949636 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 824/1621] 10.222.0.50 () {60 vars in 1153 bytes} [Tue Sep 13 16:06:50 2022] POST /identity/v3/projects => generated 405 bytes in 73 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 16:06:50.950690 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18450311-b436-415f-b917-f4d35c165e76 admin admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:06:50.960011 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.auth.core [None req-27d67d1e-2b06-48ce-9194-3332d9b9cef5 None None] MFA Rules not processed for user `4e039b46c5e3445bbb39904b00c5430e`. Rule list: `[]` (Enabled: `True`). {{(pid=81438) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:06:50.961907 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-27d67d1e-2b06-48ce-9194-3332d9b9cef5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:06:51.036162 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18450311-b436-415f-b917-f4d35c165e76 admin admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:06:51.040574 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d123848-67a8-45cb-a302-5472e5839689 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f436f07939d4855a21b195bc7566244', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:06:51.041324 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d123848-67a8-45cb-a302-5472e5839689 service neutron] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:06:51.041708 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d123848-67a8-45cb-a302-5472e5839689 service neutron] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:06:51.042088 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d123848-67a8-45cb-a302-5472e5839689 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:06:51.053439 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:06:51.077077 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 798/1622] 10.222.0.50 () {58 vars in 1009 bytes} [Tue Sep 13 16:06:50 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 143 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:06:51.079764 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11cf07f6-300b-4767-9f29-1b8494754841 None None] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:06:51.080620 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11cf07f6-300b-4767-9f29-1b8494754841 None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:06:51.080929 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11cf07f6-300b-4767-9f29-1b8494754841 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:06:51.112539 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.auth.core [None req-11cf07f6-300b-4767-9f29-1b8494754841 None None] MFA Rules not processed for user `4e039b46c5e3445bbb39904b00c5430e`. Rule list: `[]` (Enabled: `True`). {{(pid=81438) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:06:51.113966 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-11cf07f6-300b-4767-9f29-1b8494754841 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:06:51.125173 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 799/1623] 10.222.0.50 () {58 vars in 1059 bytes} [Tue Sep 13 16:06:51 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 47 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:06:51.126331 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11cf07f6-300b-4767-9f29-1b8494754841 None None] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:06:51.142144 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11cf07f6-300b-4767-9f29-1b8494754841 None None] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:06:51.146155 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96212fce-339c-4da8-bd7d-fcc35cf1ff5e service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f436f07939d4855a21b195bc7566244', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:06:51.147669 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96212fce-339c-4da8-bd7d-fcc35cf1ff5e service neutron] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:06:51.148154 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96212fce-339c-4da8-bd7d-fcc35cf1ff5e service neutron] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:06:51.148603 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96212fce-339c-4da8-bd7d-fcc35cf1ff5e service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:06:51.178130 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d123848-67a8-45cb-a302-5472e5839689 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:06:51.180035 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d123848-67a8-45cb-a302-5472e5839689 service neutron] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:06:51.193083 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 825/1624] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 16:06:50 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 243 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 16:06:51.195380 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d123848-67a8-45cb-a302-5472e5839689 service neutron] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:06:51.210033 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d123848-67a8-45cb-a302-5472e5839689 service neutron] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:06:51.223581 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed82ea7b-9f4b-4ca7-9f5d-ee875c1d52d9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:06:51.228390 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed82ea7b-9f4b-4ca7-9f5d-ee875c1d52d9 admin admin] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:06:51.229030 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed82ea7b-9f4b-4ca7-9f5d-ee875c1d52d9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:06:51.229359 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed82ea7b-9f4b-4ca7-9f5d-ee875c1d52d9 admin admin] PATH_INFO: `/v3/projects` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:06:51.230694 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed82ea7b-9f4b-4ca7-9f5d-ee875c1d52d9 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:06:51.235598 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed82ea7b-9f4b-4ca7-9f5d-ee875c1d52d9 admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:06:51.256783 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:06:51.257713 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96212fce-339c-4da8-bd7d-fcc35cf1ff5e service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:06:51.260051 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96212fce-339c-4da8-bd7d-fcc35cf1ff5e service neutron] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:06:51.275239 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 800/1625] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 16:06:51 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 150 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 16:06:51.276406 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96212fce-339c-4da8-bd7d-fcc35cf1ff5e service neutron] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:06:51.277322 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 826/1626] 10.222.0.50 () {60 vars in 1153 bytes} [Tue Sep 13 16:06:51 2022] POST /identity/v3/projects => generated 405 bytes in 83 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 16:06:51.278230 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed82ea7b-9f4b-4ca7-9f5d-ee875c1d52d9 admin admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:06:51.296268 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96212fce-339c-4da8-bd7d-fcc35cf1ff5e service neutron] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:06:51.375439 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed82ea7b-9f4b-4ca7-9f5d-ee875c1d52d9 admin admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:06:51.380176 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6c27751-9483-49f5-90d2-f813846d2a0d service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7fd6b5afd67b48f9b0f0597ccd4cf4df', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:06:51.381586 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6c27751-9483-49f5-90d2-f813846d2a0d service cinder] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:06:51.382364 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6c27751-9483-49f5-90d2-f813846d2a0d service cinder] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:06:51.382883 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6c27751-9483-49f5-90d2-f813846d2a0d service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:06:51.396845 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9459cf48-8721-446f-8bc5-97f74fb66595 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:06:51.397652 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9459cf48-8721-446f-8bc5-97f74fb66595 admin admin] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:06:51.398060 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9459cf48-8721-446f-8bc5-97f74fb66595 admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:06:51.398444 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9459cf48-8721-446f-8bc5-97f74fb66595 admin admin] PATH_INFO: `/v3/users` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:06:51.399325 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9459cf48-8721-446f-8bc5-97f74fb66595 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:06:51.401769 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9459cf48-8721-446f-8bc5-97f74fb66595 admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:06:51.431715 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 801/1627] 10.222.0.50 () {60 vars in 1144 bytes} [Tue Sep 13 16:06:51 2022] POST /identity/v3/users => generated 309 bytes in 156 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 16:06:51.432776 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9459cf48-8721-446f-8bc5-97f74fb66595 admin admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:06:51.537496 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9459cf48-8721-446f-8bc5-97f74fb66595 admin admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:06:51.544334 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6c27751-9483-49f5-90d2-f813846d2a0d service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:06:51.545008 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf0b0e46-7b34-4a00-83d0-2a32fe9cc46e service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4e039b46c5e3445bbb39904b00c5430e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'admin', 'ResellerAdmin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:06:51.545962 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf0b0e46-7b34-4a00-83d0-2a32fe9cc46e service nova] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:06:51.546119 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6c27751-9483-49f5-90d2-f813846d2a0d service cinder] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:06:51.546259 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf0b0e46-7b34-4a00-83d0-2a32fe9cc46e service nova] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:06:51.546527 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf0b0e46-7b34-4a00-83d0-2a32fe9cc46e service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:06:51.564641 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:06:51.579272 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 827/1628] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 16:06:51 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 301 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 16:06:51.580597 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6c27751-9483-49f5-90d2-f813846d2a0d service cinder] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:06:51.605206 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6c27751-9483-49f5-90d2-f813846d2a0d service cinder] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:06:51.618683 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abe5f7bc-6f17-4df8-9090-fb71fc94c9ca admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:06:51.618683 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abe5f7bc-6f17-4df8-9090-fb71fc94c9ca admin admin] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:06:51.619011 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abe5f7bc-6f17-4df8-9090-fb71fc94c9ca admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:06:51.619221 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abe5f7bc-6f17-4df8-9090-fb71fc94c9ca admin admin] PATH_INFO: `/v3/users` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:06:51.621832 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abe5f7bc-6f17-4df8-9090-fb71fc94c9ca admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:06:51.625650 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abe5f7bc-6f17-4df8-9090-fb71fc94c9ca admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:06:51.642515 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf0b0e46-7b34-4a00-83d0-2a32fe9cc46e service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:06:51.644745 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf0b0e46-7b34-4a00-83d0-2a32fe9cc46e service nova] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:06:51.660673 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 802/1629] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 16:06:51 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 228 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 16:06:51.661964 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf0b0e46-7b34-4a00-83d0-2a32fe9cc46e service nova] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:06:51.672149 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 828/1630] 10.222.0.50 () {60 vars in 1144 bytes} [Tue Sep 13 16:06:51 2022] POST /identity/v3/users => generated 309 bytes in 92 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 16:06:51.676341 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf0b0e46-7b34-4a00-83d0-2a32fe9cc46e service nova] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:06:51.679226 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abe5f7bc-6f17-4df8-9090-fb71fc94c9ca admin admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:06:51.682556 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a672358d-1bff-4545-9b3a-1930a910bb12 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:06:51.683288 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a672358d-1bff-4545-9b3a-1930a910bb12 admin admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:06:51.683796 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a672358d-1bff-4545-9b3a-1930a910bb12 admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:06:51.684213 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a672358d-1bff-4545-9b3a-1930a910bb12 admin admin] PATH_INFO: `/v3/roles` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:06:51.685205 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a672358d-1bff-4545-9b3a-1930a910bb12 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:06:51.687779 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a672358d-1bff-4545-9b3a-1930a910bb12 admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:06:51.700706 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 803/1631] 10.222.0.50 () {58 vars in 1122 bytes} [Tue Sep 13 16:06:51 2022] GET /identity/v3/roles => generated 1420 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 16:06:51.703456 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abe5f7bc-6f17-4df8-9090-fb71fc94c9ca admin admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:06:51.705363 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a672358d-1bff-4545-9b3a-1930a910bb12 admin admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:06:51.717953 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a672358d-1bff-4545-9b3a-1930a910bb12 admin admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:06:51.719303 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4879f9e4-aad2-4ded-bf31-344eda22ccae admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:06:51.720437 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4879f9e4-aad2-4ded-bf31-344eda22ccae admin admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:06:51.720437 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4879f9e4-aad2-4ded-bf31-344eda22ccae admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:06:51.720717 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4879f9e4-aad2-4ded-bf31-344eda22ccae admin admin] PATH_INFO: `/v3/roles` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:06:51.722424 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4879f9e4-aad2-4ded-bf31-344eda22ccae admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:06:51.725237 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-915d401d-322a-4a87-b001-9033c849e091 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:06:51.726205 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-915d401d-322a-4a87-b001-9033c849e091 admin admin] REQUEST_METHOD: `PUT` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:06:51.726677 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4879f9e4-aad2-4ded-bf31-344eda22ccae admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:06:51.735253 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-915d401d-322a-4a87-b001-9033c849e091 admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:06:51.735594 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-915d401d-322a-4a87-b001-9033c849e091 admin admin] PATH_INFO: `/v3/projects/936eda6390354d37b998e2268154fd6e/users/80840071f51f4870977c3e4e481ceae7/roles/031d4e1a23e34f71858fc4a6dfbd4ccb` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:06:51.754877 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 829/1632] 10.222.0.50 () {58 vars in 1122 bytes} [Tue Sep 13 16:06:51 2022] GET /identity/v3/roles => generated 1420 bytes in 77 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 16:06:51.755981 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4879f9e4-aad2-4ded-bf31-344eda22ccae admin admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:06:51.765046 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-915d401d-322a-4a87-b001-9033c849e091 admin admin] RBAC: Authorizing `identity:create_grant(project_id=936eda6390354d37b998e2268154fd6e, user_id=80840071f51f4870977c3e4e481ceae7, role_id=031d4e1a23e34f71858fc4a6dfbd4ccb)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:06:51.771457 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-915d401d-322a-4a87-b001-9033c849e091 admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:06:51.780217 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4879f9e4-aad2-4ded-bf31-344eda22ccae admin admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:06:51.794196 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 804/1633] 10.222.0.50 () {60 vars in 1483 bytes} [Tue Sep 13 16:06:51 2022] PUT /identity/v3/projects/936eda6390354d37b998e2268154fd6e/users/80840071f51f4870977c3e4e481ceae7/roles/031d4e1a23e34f71858fc4a6dfbd4ccb => generated 0 bytes in 89 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 13 16:06:51.794961 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b7f5038-ec2a-4b87-a82d-4aa5aad01af9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:06:51.795657 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b7f5038-ec2a-4b87-a82d-4aa5aad01af9 admin admin] REQUEST_METHOD: `PUT` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:06:51.796389 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b7f5038-ec2a-4b87-a82d-4aa5aad01af9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:06:51.796389 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b7f5038-ec2a-4b87-a82d-4aa5aad01af9 admin admin] PATH_INFO: `/v3/projects/b8139ba8cf42465bad4369b51591a32e/users/35aa294e5c9944de9550bf9ce05698fb/roles/031d4e1a23e34f71858fc4a6dfbd4ccb` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:06:51.800461 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-915d401d-322a-4a87-b001-9033c849e091 admin admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:06:51.836071 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b7f5038-ec2a-4b87-a82d-4aa5aad01af9 admin admin] RBAC: Authorizing `identity:create_grant(project_id=b8139ba8cf42465bad4369b51591a32e, user_id=35aa294e5c9944de9550bf9ce05698fb, role_id=031d4e1a23e34f71858fc4a6dfbd4ccb)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:06:51.845628 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b7f5038-ec2a-4b87-a82d-4aa5aad01af9 admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:06:51.884763 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 830/1634] 10.222.0.50 () {60 vars in 1483 bytes} [Tue Sep 13 16:06:51 2022] PUT /identity/v3/projects/b8139ba8cf42465bad4369b51591a32e/users/35aa294e5c9944de9550bf9ce05698fb/roles/031d4e1a23e34f71858fc4a6dfbd4ccb => generated 0 bytes in 126 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 16:06:51.886108 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b7f5038-ec2a-4b87-a82d-4aa5aad01af9 admin admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:06:51.920813 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-915d401d-322a-4a87-b001-9033c849e091 admin admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:06:51.980599 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:INSERT=5,keystone:SELECT=95 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:06:52.025504 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b7f5038-ec2a-4b87-a82d-4aa5aad01af9 admin admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:06:52.037671 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8155db27-3b7f-4850-b2f4-aac298681b94 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:06:52.038447 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8155db27-3b7f-4850-b2f4-aac298681b94 admin admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:06:52.039935 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8155db27-3b7f-4850-b2f4-aac298681b94 admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:06:52.040328 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8155db27-3b7f-4850-b2f4-aac298681b94 admin admin] PATH_INFO: `/v3/roles` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:06:52.041295 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8155db27-3b7f-4850-b2f4-aac298681b94 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:06:52.046281 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8155db27-3b7f-4850-b2f4-aac298681b94 admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:06:52.062318 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5345f45e-a6fc-4e18-9906-769203a2f072 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:06:52.066114 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5345f45e-a6fc-4e18-9906-769203a2f072 admin admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:06:52.067234 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5345f45e-a6fc-4e18-9906-769203a2f072 admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:06:52.067668 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5345f45e-a6fc-4e18-9906-769203a2f072 admin admin] PATH_INFO: `/v3/roles` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:06:52.070331 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 831/1635] 10.222.0.50 () {58 vars in 1122 bytes} [Tue Sep 13 16:06:51 2022] GET /identity/v3/roles => generated 1420 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 16:06:52.074933 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5345f45e-a6fc-4e18-9906-769203a2f072 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:06:52.077804 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5345f45e-a6fc-4e18-9906-769203a2f072 admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:06:52.079635 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8155db27-3b7f-4850-b2f4-aac298681b94 admin admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:06:52.101626 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 805/1636] 10.222.0.50 () {58 vars in 1122 bytes} [Tue Sep 13 16:06:51 2022] GET /identity/v3/roles => generated 1420 bytes in 304 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 13 16:06:52.102469 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8155db27-3b7f-4850-b2f4-aac298681b94 admin admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:06:52.105943 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5345f45e-a6fc-4e18-9906-769203a2f072 admin admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:06:52.116372 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96e4f0d0-3ce5-459d-9fc0-138c44e47b34 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:06:52.117356 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96e4f0d0-3ce5-459d-9fc0-138c44e47b34 admin admin] REQUEST_METHOD: `PUT` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:06:52.117741 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96e4f0d0-3ce5-459d-9fc0-138c44e47b34 admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:06:52.118118 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96e4f0d0-3ce5-459d-9fc0-138c44e47b34 admin admin] PATH_INFO: `/v3/projects/b8139ba8cf42465bad4369b51591a32e/users/35aa294e5c9944de9550bf9ce05698fb/roles/031d4e1a23e34f71858fc4a6dfbd4ccb` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:06:52.121690 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96e4f0d0-3ce5-459d-9fc0-138c44e47b34 admin admin] RBAC: Authorizing `identity:create_grant(project_id=b8139ba8cf42465bad4369b51591a32e, user_id=35aa294e5c9944de9550bf9ce05698fb, role_id=031d4e1a23e34f71858fc4a6dfbd4ccb)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:06:52.127516 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5345f45e-a6fc-4e18-9906-769203a2f072 admin admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:06:52.129764 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96e4f0d0-3ce5-459d-9fc0-138c44e47b34 admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:06:52.139565 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-725ea16a-71d9-4890-8eb0-d61cd65e6ee4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:06:52.140384 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-725ea16a-71d9-4890-8eb0-d61cd65e6ee4 admin admin] REQUEST_METHOD: `PUT` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:06:52.140763 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-725ea16a-71d9-4890-8eb0-d61cd65e6ee4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:06:52.141661 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-725ea16a-71d9-4890-8eb0-d61cd65e6ee4 admin admin] PATH_INFO: `/v3/projects/936eda6390354d37b998e2268154fd6e/users/80840071f51f4870977c3e4e481ceae7/roles/031d4e1a23e34f71858fc4a6dfbd4ccb` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:06:52.146209 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-725ea16a-71d9-4890-8eb0-d61cd65e6ee4 admin admin] RBAC: Authorizing `identity:create_grant(project_id=936eda6390354d37b998e2268154fd6e, user_id=80840071f51f4870977c3e4e481ceae7, role_id=031d4e1a23e34f71858fc4a6dfbd4ccb)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:06:52.150978 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-725ea16a-71d9-4890-8eb0-d61cd65e6ee4 admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:06:52.157093 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 832/1637] 10.222.0.50 () {60 vars in 1483 bytes} [Tue Sep 13 16:06:52 2022] PUT /identity/v3/projects/b8139ba8cf42465bad4369b51591a32e/users/35aa294e5c9944de9550bf9ce05698fb/roles/031d4e1a23e34f71858fc4a6dfbd4ccb => generated 0 bytes in 78 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 16:06:52.161939 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13d52cf6-a7f0-4cd5-99a3-c9acf1944746 None None] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:06:52.163450 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13d52cf6-a7f0-4cd5-99a3-c9acf1944746 None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:06:52.164912 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13d52cf6-a7f0-4cd5-99a3-c9acf1944746 None None] PATH_INFO: `/` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:06:52.169279 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 833/1638] 10.222.0.50 () {54 vars in 912 bytes} [Tue Sep 13 16:06:52 2022] GET /identity => generated 270 bytes in 12 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 16:06:52.175801 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-714f68e2-261a-45d6-8c08-0cfb9c635248 None None] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:06:52.176208 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-714f68e2-261a-45d6-8c08-0cfb9c635248 None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:06:52.176583 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-714f68e2-261a-45d6-8c08-0cfb9c635248 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:06:52.183271 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 806/1639] 10.222.0.50 () {60 vars in 1483 bytes} [Tue Sep 13 16:06:52 2022] PUT /identity/v3/projects/936eda6390354d37b998e2268154fd6e/users/80840071f51f4870977c3e4e481ceae7/roles/031d4e1a23e34f71858fc4a6dfbd4ccb => generated 0 bytes in 78 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 16:06:52.195240 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efb57034-268a-4292-a1e2-ee9adb345897 None None] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:06:52.196582 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efb57034-268a-4292-a1e2-ee9adb345897 None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:06:52.196976 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efb57034-268a-4292-a1e2-ee9adb345897 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:06:52.210825 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.auth.core [None req-714f68e2-261a-45d6-8c08-0cfb9c635248 None None] MFA Rules not processed for user `4e039b46c5e3445bbb39904b00c5430e`. Rule list: `[]` (Enabled: `True`). {{(pid=81437) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:06:52.212095 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-714f68e2-261a-45d6-8c08-0cfb9c635248 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:06:52.253185 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:06:52.276841 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.auth.core [None req-efb57034-268a-4292-a1e2-ee9adb345897 None None] MFA Rules not processed for user `80840071f51f4870977c3e4e481ceae7`. Rule list: `[]` (Enabled: `True`). {{(pid=81438) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:06:52.285196 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-efb57034-268a-4292-a1e2-ee9adb345897 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:06:52.365558 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 834/1640] 10.222.0.50 () {58 vars in 1009 bytes} [Tue Sep 13 16:06:52 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 190 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:06:52.379341 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-714f68e2-261a-45d6-8c08-0cfb9c635248 None None] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:06:52.455436 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 807/1641] 10.222.0.50 () {58 vars in 958 bytes} [Tue Sep 13 16:06:52 2022] POST /identity/v3/auth/tokens => generated 3349 bytes in 262 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:06:52.463037 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-714f68e2-261a-45d6-8c08-0cfb9c635248 None None] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:06:52.469731 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28b630e1-bef8-44d3-94a0-1ae4d39c0839 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7fd6b5afd67b48f9b0f0597ccd4cf4df', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:06:52.470977 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28b630e1-bef8-44d3-94a0-1ae4d39c0839 service cinder] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:06:52.471394 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28b630e1-bef8-44d3-94a0-1ae4d39c0839 service cinder] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:06:52.471835 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28b630e1-bef8-44d3-94a0-1ae4d39c0839 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:06:52.492423 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28b630e1-bef8-44d3-94a0-1ae4d39c0839 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:06:52.494294 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28b630e1-bef8-44d3-94a0-1ae4d39c0839 service cinder] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:06:52.508776 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 835/1642] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 16:06:52 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 130 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 13 16:06:52.746110 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efb57034-268a-4292-a1e2-ee9adb345897 None None] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:06:52.758734 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efb57034-268a-4292-a1e2-ee9adb345897 None None] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:06:52.765903 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72a85a32-7da4-4c30-885d-9f3133230f26 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7fd6b5afd67b48f9b0f0597ccd4cf4df', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:06:52.767007 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72a85a32-7da4-4c30-885d-9f3133230f26 service cinder] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:06:52.767443 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72a85a32-7da4-4c30-885d-9f3133230f26 service cinder] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:06:52.767988 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72a85a32-7da4-4c30-885d-9f3133230f26 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:06:52.781527 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72a85a32-7da4-4c30-885d-9f3133230f26 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:06:52.783406 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72a85a32-7da4-4c30-885d-9f3133230f26 service cinder] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:06:52.800323 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 808/1643] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 16:06:52 2022] GET /identity/v3/auth/tokens => generated 3349 bytes in 55 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 16:06:53.575442 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28b630e1-bef8-44d3-94a0-1ae4d39c0839 service cinder] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:06:53.727196 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28b630e1-bef8-44d3-94a0-1ae4d39c0839 service cinder] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:06:53.733126 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f57e430-6079-451d-b225-177db30cce0b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:06:53.733978 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f57e430-6079-451d-b225-177db30cce0b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:06:53.734136 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f57e430-6079-451d-b225-177db30cce0b admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:06:53.734509 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f57e430-6079-451d-b225-177db30cce0b admin admin] PATH_INFO: `/v3/users/6f358deec606446a9fc085dcd4817d08` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:06:53.735786 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f57e430-6079-451d-b225-177db30cce0b admin admin] RBAC: Authorizing `identity:delete_user(user_id=6f358deec606446a9fc085dcd4817d08)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:06:53.736891 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: WARNING py.warnings [None req-2f57e430-6079-451d-b225-177db30cce0b admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:delete_user": "(role:admin and system_scope:all) or (role:admin and token.domain.id:%(target.user.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Sep 13 16:06:53.736891 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: warnings.warn(msg) Sep 13 16:06:53.736891 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: Sep 13 16:06:53.738289 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f57e430-6079-451d-b225-177db30cce0b admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:06:53.740257 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:06:53.853389 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-2f57e430-6079-451d-b225-177db30cce0b admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '6f358deec606446a9fc085dcd4817d08'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:06:53.867165 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-2f57e430-6079-451d-b225-177db30cce0b admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '6f358deec606446a9fc085dcd4817d08'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:06:53.966171 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-2f57e430-6079-451d-b225-177db30cce0b admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '6f358deec606446a9fc085dcd4817d08'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:06:53.966171 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-2f57e430-6079-451d-b225-177db30cce0b admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '6f358deec606446a9fc085dcd4817d08'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:06:53.980831 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-2f57e430-6079-451d-b225-177db30cce0b admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '6f358deec606446a9fc085dcd4817d08'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:06:53.994627 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 836/1644] 10.222.0.50 () {58 vars in 1224 bytes} [Tue Sep 13 16:06:53 2022] DELETE /identity/v3/users/6f358deec606446a9fc085dcd4817d08 => generated 0 bytes in 416 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 16:06:53.995950 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec707c70-582f-4bde-b050-f5caac4ac411 None None] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:06:53.996490 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec707c70-582f-4bde-b050-f5caac4ac411 None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:06:53.996607 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec707c70-582f-4bde-b050-f5caac4ac411 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:06:54.020973 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.auth.core [None req-ec707c70-582f-4bde-b050-f5caac4ac411 None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81438) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:06:54.022865 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-ec707c70-582f-4bde-b050-f5caac4ac411 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:06:54.151686 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 809/1645] 10.222.0.50 () {58 vars in 958 bytes} [Tue Sep 13 16:06:53 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 157 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:06:54.178716 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f57e430-6079-451d-b225-177db30cce0b admin admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:06:54.180418 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-2f57e430-6079-451d-b225-177db30cce0b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:06:54.300534 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f57e430-6079-451d-b225-177db30cce0b admin admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:06:54.306634 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c89abb7-5c6d-48a2-bac6-3c89e0f91c65 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f436f07939d4855a21b195bc7566244', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:06:54.307485 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c89abb7-5c6d-48a2-bac6-3c89e0f91c65 service neutron] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:06:54.307834 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c89abb7-5c6d-48a2-bac6-3c89e0f91c65 service neutron] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:06:54.308119 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c89abb7-5c6d-48a2-bac6-3c89e0f91c65 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:06:54.344553 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c89abb7-5c6d-48a2-bac6-3c89e0f91c65 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:06:54.354126 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c89abb7-5c6d-48a2-bac6-3c89e0f91c65 service neutron] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:06:54.399891 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 837/1646] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 16:06:54 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 209 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 16:06:55.956226 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec707c70-582f-4bde-b050-f5caac4ac411 None None] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:06:55.958618 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-ec707c70-582f-4bde-b050-f5caac4ac411 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:06:55.975739 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec707c70-582f-4bde-b050-f5caac4ac411 None None] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:06:55.981307 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfb5a67e-45ed-4d12-a55e-951c42af14d8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:06:55.982312 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfb5a67e-45ed-4d12-a55e-951c42af14d8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:06:55.982861 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfb5a67e-45ed-4d12-a55e-951c42af14d8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:06:55.983365 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfb5a67e-45ed-4d12-a55e-951c42af14d8 admin admin] PATH_INFO: `/v3/users/ff06e46c42034f59b393ce213830cfa5` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:06:55.985162 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfb5a67e-45ed-4d12-a55e-951c42af14d8 admin admin] RBAC: Authorizing `identity:delete_user(user_id=ff06e46c42034f59b393ce213830cfa5)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:06:55.988975 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfb5a67e-45ed-4d12-a55e-951c42af14d8 admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:06:56.071540 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-bfb5a67e-45ed-4d12-a55e-951c42af14d8 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'ff06e46c42034f59b393ce213830cfa5'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:06:56.084638 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-bfb5a67e-45ed-4d12-a55e-951c42af14d8 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'ff06e46c42034f59b393ce213830cfa5'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:06:56.104024 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-bfb5a67e-45ed-4d12-a55e-951c42af14d8 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'ff06e46c42034f59b393ce213830cfa5'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:06:56.105087 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-bfb5a67e-45ed-4d12-a55e-951c42af14d8 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'ff06e46c42034f59b393ce213830cfa5'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:06:56.112543 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:06:56.118570 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-bfb5a67e-45ed-4d12-a55e-951c42af14d8 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'ff06e46c42034f59b393ce213830cfa5'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:06:56.133820 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 810/1647] 10.222.0.50 () {58 vars in 1224 bytes} [Tue Sep 13 16:06:55 2022] DELETE /identity/v3/users/ff06e46c42034f59b393ce213830cfa5 => generated 0 bytes in 177 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 16:06:56.135755 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de15f3be-526b-47b8-8ad7-79181be78109 None None] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:06:56.136344 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de15f3be-526b-47b8-8ad7-79181be78109 None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:06:56.136674 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de15f3be-526b-47b8-8ad7-79181be78109 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:06:56.170463 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.auth.core [None req-de15f3be-526b-47b8-8ad7-79181be78109 None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81437) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:06:56.173645 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-de15f3be-526b-47b8-8ad7-79181be78109 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:06:56.310949 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 838/1648] 10.222.0.50 () {58 vars in 958 bytes} [Tue Sep 13 16:06:56 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 176 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:06:56.366570 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5e291d3-aedb-4ce5-a22d-b24d633c7054 None None] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:06:56.366570 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5e291d3-aedb-4ce5-a22d-b24d633c7054 None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:06:56.366570 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5e291d3-aedb-4ce5-a22d-b24d633c7054 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:06:56.379366 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de15f3be-526b-47b8-8ad7-79181be78109 None None] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:06:56.386123 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-de15f3be-526b-47b8-8ad7-79181be78109 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:06:56.422191 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.auth.core [None req-e5e291d3-aedb-4ce5-a22d-b24d633c7054 None None] MFA Rules not processed for user `35aa294e5c9944de9550bf9ce05698fb`. Rule list: `[]` (Enabled: `True`). {{(pid=81438) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:06:56.423460 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-e5e291d3-aedb-4ce5-a22d-b24d633c7054 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:06:56.432441 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:06:56.514133 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de15f3be-526b-47b8-8ad7-79181be78109 None None] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:06:56.521706 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ffbb94f-7d90-446b-86aa-3870b5c192b3 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f436f07939d4855a21b195bc7566244', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:06:56.522145 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ffbb94f-7d90-446b-86aa-3870b5c192b3 service neutron] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:06:56.522986 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ffbb94f-7d90-446b-86aa-3870b5c192b3 service neutron] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:06:56.522986 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ffbb94f-7d90-446b-86aa-3870b5c192b3 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:06:56.538568 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ffbb94f-7d90-446b-86aa-3870b5c192b3 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:06:56.540057 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ffbb94f-7d90-446b-86aa-3870b5c192b3 service neutron] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:06:56.563683 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 839/1649] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 16:06:56 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 193 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 16:06:56.580022 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 811/1650] 10.222.0.50 () {58 vars in 958 bytes} [Tue Sep 13 16:06:56 2022] POST /identity/v3/auth/tokens => generated 3349 bytes in 215 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:06:56.844321 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ffbb94f-7d90-446b-86aa-3870b5c192b3 service neutron] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:06:56.845778 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-6ffbb94f-7d90-446b-86aa-3870b5c192b3 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:06:56.883337 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ffbb94f-7d90-446b-86aa-3870b5c192b3 service neutron] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:06:56.905446 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3c6f339-50ad-426a-9ed7-05ed6c100994 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:06:56.906172 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3c6f339-50ad-426a-9ed7-05ed6c100994 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:06:56.906457 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3c6f339-50ad-426a-9ed7-05ed6c100994 admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:06:56.908127 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3c6f339-50ad-426a-9ed7-05ed6c100994 admin admin] PATH_INFO: `/v3/projects/3a13f08db2354b399d2be343bb888d3f` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:06:56.910574 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3c6f339-50ad-426a-9ed7-05ed6c100994 admin admin] RBAC: Authorizing `identity:delete_project(project_id=3a13f08db2354b399d2be343bb888d3f)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:06:56.917351 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3c6f339-50ad-426a-9ed7-05ed6c100994 admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:06:57.058892 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-e3c6f339-50ad-426a-9ed7-05ed6c100994 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '3a13f08db2354b399d2be343bb888d3f', 'request_id': 'req-e3c6f339-50ad-426a-9ed7-05ed6c100994'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:06:57.082537 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-e3c6f339-50ad-426a-9ed7-05ed6c100994 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '3a13f08db2354b399d2be343bb888d3f', 'request_id': 'req-e3c6f339-50ad-426a-9ed7-05ed6c100994'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:06:57.092502 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-e3c6f339-50ad-426a-9ed7-05ed6c100994 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '3a13f08db2354b399d2be343bb888d3f', 'request_id': 'req-e3c6f339-50ad-426a-9ed7-05ed6c100994'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:06:57.109706 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-e3c6f339-50ad-426a-9ed7-05ed6c100994 admin admin] The token cache is being invalidate because project 3a13f08db2354b399d2be343bb888d3f was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81437) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 13 16:06:57.110058 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-e3c6f339-50ad-426a-9ed7-05ed6c100994 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:06:57.110766 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-e3c6f339-50ad-426a-9ed7-05ed6c100994 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:06:57.116101 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 840/1651] 10.222.0.50 () {58 vars in 1233 bytes} [Tue Sep 13 16:06:56 2022] DELETE /identity/v3/projects/3a13f08db2354b399d2be343bb888d3f => generated 0 bytes in 269 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 16:06:57.116820 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5e291d3-aedb-4ce5-a22d-b24d633c7054 None None] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:06:57.142645 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5e291d3-aedb-4ce5-a22d-b24d633c7054 None None] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:06:57.410929 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5ec0f5e-99ee-4c83-8085-c2f31b1ebce8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:06:57.414911 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5ec0f5e-99ee-4c83-8085-c2f31b1ebce8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:06:57.417208 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5ec0f5e-99ee-4c83-8085-c2f31b1ebce8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:06:57.424155 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5ec0f5e-99ee-4c83-8085-c2f31b1ebce8 admin admin] PATH_INFO: `/v3/users/80840071f51f4870977c3e4e481ceae7` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:06:57.430850 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5ec0f5e-99ee-4c83-8085-c2f31b1ebce8 admin admin] RBAC: Authorizing `identity:delete_user(user_id=80840071f51f4870977c3e4e481ceae7)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:06:57.432960 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5ec0f5e-99ee-4c83-8085-c2f31b1ebce8 admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:06:57.592319 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-d5ec0f5e-99ee-4c83-8085-c2f31b1ebce8 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '80840071f51f4870977c3e4e481ceae7'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:06:57.611982 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-d5ec0f5e-99ee-4c83-8085-c2f31b1ebce8 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '80840071f51f4870977c3e4e481ceae7'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:06:57.647376 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-d5ec0f5e-99ee-4c83-8085-c2f31b1ebce8 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '80840071f51f4870977c3e4e481ceae7'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:06:57.648236 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-d5ec0f5e-99ee-4c83-8085-c2f31b1ebce8 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '80840071f51f4870977c3e4e481ceae7'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:06:57.665898 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-d5ec0f5e-99ee-4c83-8085-c2f31b1ebce8 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '80840071f51f4870977c3e4e481ceae7'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:06:57.669050 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:06:57.679535 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 812/1652] 10.222.0.50 () {58 vars in 1224 bytes} [Tue Sep 13 16:06:57 2022] DELETE /identity/v3/users/80840071f51f4870977c3e4e481ceae7 => generated 0 bytes in 561 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 16:06:58.177402 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3c6f339-50ad-426a-9ed7-05ed6c100994 admin admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:06:58.183254 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-e3c6f339-50ad-426a-9ed7-05ed6c100994 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:06:58.224048 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3c6f339-50ad-426a-9ed7-05ed6c100994 admin admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:06:58.382630 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b76bba7c-ab81-41be-9b5c-3ed6ed8d0f38 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:06:58.383531 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b76bba7c-ab81-41be-9b5c-3ed6ed8d0f38 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:06:58.383884 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b76bba7c-ab81-41be-9b5c-3ed6ed8d0f38 admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:06:58.384238 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b76bba7c-ab81-41be-9b5c-3ed6ed8d0f38 admin admin] PATH_INFO: `/v3/projects/8be8793418084985bce6350626212347` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:06:58.385491 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b76bba7c-ab81-41be-9b5c-3ed6ed8d0f38 admin admin] RBAC: Authorizing `identity:delete_project(project_id=8be8793418084985bce6350626212347)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:06:58.387929 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b76bba7c-ab81-41be-9b5c-3ed6ed8d0f38 admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:06:58.500258 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-b76bba7c-ab81-41be-9b5c-3ed6ed8d0f38 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '8be8793418084985bce6350626212347', 'request_id': 'req-b76bba7c-ab81-41be-9b5c-3ed6ed8d0f38'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:06:58.502664 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=88,keystone:DELETE=11,keystone:INSERT=1 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:06:58.526503 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-b76bba7c-ab81-41be-9b5c-3ed6ed8d0f38 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '8be8793418084985bce6350626212347', 'request_id': 'req-b76bba7c-ab81-41be-9b5c-3ed6ed8d0f38'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:06:58.534835 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-b76bba7c-ab81-41be-9b5c-3ed6ed8d0f38 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '8be8793418084985bce6350626212347', 'request_id': 'req-b76bba7c-ab81-41be-9b5c-3ed6ed8d0f38'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:06:58.564795 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-b76bba7c-ab81-41be-9b5c-3ed6ed8d0f38 admin admin] The token cache is being invalidate because project 8be8793418084985bce6350626212347 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81437) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 13 16:06:58.565281 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-b76bba7c-ab81-41be-9b5c-3ed6ed8d0f38 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:06:58.567087 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-b76bba7c-ab81-41be-9b5c-3ed6ed8d0f38 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:06:58.569406 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 841/1653] 10.222.0.50 () {58 vars in 1233 bytes} [Tue Sep 13 16:06:58 2022] DELETE /identity/v3/projects/8be8793418084985bce6350626212347 => generated 0 bytes in 394 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 16:06:58.574008 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2aaad71e-c057-41f9-b5f1-c87e51cbc4f2 None None] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:06:58.575050 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2aaad71e-c057-41f9-b5f1-c87e51cbc4f2 None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:06:58.575563 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2aaad71e-c057-41f9-b5f1-c87e51cbc4f2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:06:58.619591 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.auth.core [None req-2aaad71e-c057-41f9-b5f1-c87e51cbc4f2 None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81438) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:06:58.621269 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-2aaad71e-c057-41f9-b5f1-c87e51cbc4f2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:06:58.759151 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 813/1654] 10.222.0.50 () {58 vars in 958 bytes} [Tue Sep 13 16:06:58 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 186 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:06:58.768416 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-563a2964-8832-4d64-83ed-bf32f94c9da8 None None] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:06:58.768701 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-563a2964-8832-4d64-83ed-bf32f94c9da8 None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:06:58.768980 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-563a2964-8832-4d64-83ed-bf32f94c9da8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:06:58.807172 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.auth.core [None req-563a2964-8832-4d64-83ed-bf32f94c9da8 None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81437) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:06:58.808488 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-563a2964-8832-4d64-83ed-bf32f94c9da8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:06:58.831854 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 842/1655] 10.222.0.50 () {58 vars in 958 bytes} [Tue Sep 13 16:06:58 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 66 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:06:58.832510 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2aaad71e-c057-41f9-b5f1-c87e51cbc4f2 None None] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:06:58.850689 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2aaad71e-c057-41f9-b5f1-c87e51cbc4f2 None None] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:06:58.881460 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff4968b1-17d1-4a02-8a84-1e10499804d5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:06:58.882095 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff4968b1-17d1-4a02-8a84-1e10499804d5 admin admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:06:58.882373 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff4968b1-17d1-4a02-8a84-1e10499804d5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:06:58.882643 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff4968b1-17d1-4a02-8a84-1e10499804d5 admin admin] PATH_INFO: `/v3/domains` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:06:58.888050 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff4968b1-17d1-4a02-8a84-1e10499804d5 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81438) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 16:06:58.888376 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff4968b1-17d1-4a02-8a84-1e10499804d5 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:06:58.890033 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff4968b1-17d1-4a02-8a84-1e10499804d5 admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:06:58.938390 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 814/1656] 10.222.0.50 () {58 vars in 1153 bytes} [Tue Sep 13 16:06:58 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 106 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 13 16:06:58.942353 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-563a2964-8832-4d64-83ed-bf32f94c9da8 None None] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:06:58.959194 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-563a2964-8832-4d64-83ed-bf32f94c9da8 None None] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:06:58.982742 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1ab8d4c-c85b-4b83-8dd5-efccf687acae admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:06:58.983418 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1ab8d4c-c85b-4b83-8dd5-efccf687acae admin admin] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:06:58.983695 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1ab8d4c-c85b-4b83-8dd5-efccf687acae admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:06:58.983966 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1ab8d4c-c85b-4b83-8dd5-efccf687acae admin admin] PATH_INFO: `/v3/projects` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:06:58.984611 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1ab8d4c-c85b-4b83-8dd5-efccf687acae admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:06:58.986604 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1ab8d4c-c85b-4b83-8dd5-efccf687acae admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:06:59.084503 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 843/1657] 10.222.0.50 () {60 vars in 1153 bytes} [Tue Sep 13 16:06:58 2022] POST /identity/v3/projects => generated 365 bytes in 142 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 16:06:59.088627 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff4968b1-17d1-4a02-8a84-1e10499804d5 admin admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:06:59.104864 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff4968b1-17d1-4a02-8a84-1e10499804d5 admin admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:06:59.344657 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c70f5918-5e7d-415c-91e5-ad82ebde5436 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:06:59.349899 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c70f5918-5e7d-415c-91e5-ad82ebde5436 admin admin] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:06:59.350249 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c70f5918-5e7d-415c-91e5-ad82ebde5436 admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:06:59.350941 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c70f5918-5e7d-415c-91e5-ad82ebde5436 admin admin] PATH_INFO: `/v3/users` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:06:59.351677 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c70f5918-5e7d-415c-91e5-ad82ebde5436 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:06:59.364585 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c70f5918-5e7d-415c-91e5-ad82ebde5436 admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:06:59.418117 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 815/1658] 10.222.0.50 () {60 vars in 1144 bytes} [Tue Sep 13 16:06:59 2022] POST /identity/v3/users => generated 283 bytes in 330 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 16:06:59.423796 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1ab8d4c-c85b-4b83-8dd5-efccf687acae admin admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:06:59.438340 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1ab8d4c-c85b-4b83-8dd5-efccf687acae admin admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:06:59.456742 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-489f95ff-bced-4069-8df1-ac0bc4b1ab09 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:06:59.457570 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-489f95ff-bced-4069-8df1-ac0bc4b1ab09 admin admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:06:59.457849 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-489f95ff-bced-4069-8df1-ac0bc4b1ab09 admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:06:59.458193 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-489f95ff-bced-4069-8df1-ac0bc4b1ab09 admin admin] PATH_INFO: `/v3/roles` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:06:59.458942 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-489f95ff-bced-4069-8df1-ac0bc4b1ab09 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:06:59.461336 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-489f95ff-bced-4069-8df1-ac0bc4b1ab09 admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:06:59.495239 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 844/1659] 10.222.0.50 () {58 vars in 1122 bytes} [Tue Sep 13 16:06:59 2022] GET /identity/v3/roles => generated 1420 bytes in 72 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 16:06:59.501483 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c70f5918-5e7d-415c-91e5-ad82ebde5436 admin admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:06:59.523740 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c70f5918-5e7d-415c-91e5-ad82ebde5436 admin admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:06:59.540320 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55862fe8-2262-4dad-841f-28edc125a024 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:06:59.547100 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55862fe8-2262-4dad-841f-28edc125a024 admin admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:06:59.547389 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55862fe8-2262-4dad-841f-28edc125a024 admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:06:59.547663 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55862fe8-2262-4dad-841f-28edc125a024 admin admin] PATH_INFO: `/v3/roles` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:06:59.550633 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55862fe8-2262-4dad-841f-28edc125a024 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:06:59.555059 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55862fe8-2262-4dad-841f-28edc125a024 admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:06:59.618934 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 816/1660] 10.222.0.50 () {58 vars in 1122 bytes} [Tue Sep 13 16:06:59 2022] GET /identity/v3/roles => generated 1420 bytes in 119 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 16:06:59.620349 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-489f95ff-bced-4069-8df1-ac0bc4b1ab09 admin admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:06:59.635915 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-489f95ff-bced-4069-8df1-ac0bc4b1ab09 admin admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:06:59.648853 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-010a9684-35ca-4e5b-a56a-439f843441e4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:06:59.652530 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-010a9684-35ca-4e5b-a56a-439f843441e4 admin admin] REQUEST_METHOD: `PUT` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:06:59.652837 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-010a9684-35ca-4e5b-a56a-439f843441e4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:06:59.653164 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-010a9684-35ca-4e5b-a56a-439f843441e4 admin admin] PATH_INFO: `/v3/projects/b48b1ff8f2904a37af937869f957a8c2/users/f6bfdc6bb6bf442cb43bb1d9b9093ed5/roles/15db4aaca3264ae29bde38903ba43471` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:06:59.677792 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be2a75fe-a58c-4e55-9692-70b919231641 None None] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:06:59.678099 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be2a75fe-a58c-4e55-9692-70b919231641 None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:06:59.678276 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be2a75fe-a58c-4e55-9692-70b919231641 None None] PATH_INFO: `/` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:06:59.679914 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 817/1661] 10.222.0.50 () {54 vars in 921 bytes} [Tue Sep 13 16:06:59 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 13 16:06:59.691651 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85dcb61c-bc2b-4072-b16f-ea85df78a97e None None] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:06:59.692069 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85dcb61c-bc2b-4072-b16f-ea85df78a97e None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:06:59.692255 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85dcb61c-bc2b-4072-b16f-ea85df78a97e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:06:59.695741 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-010a9684-35ca-4e5b-a56a-439f843441e4 admin admin] RBAC: Authorizing `identity:create_grant(project_id=b48b1ff8f2904a37af937869f957a8c2, user_id=f6bfdc6bb6bf442cb43bb1d9b9093ed5, role_id=15db4aaca3264ae29bde38903ba43471)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:06:59.703113 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-010a9684-35ca-4e5b-a56a-439f843441e4 admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:06:59.733499 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.auth.core [None req-85dcb61c-bc2b-4072-b16f-ea85df78a97e None None] MFA Rules not processed for user `31480b61418340bd88583f034f67412e`. Rule list: `[]` (Enabled: `True`). {{(pid=81438) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:06:59.733983 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 845/1662] 10.222.0.50 () {60 vars in 1483 bytes} [Tue Sep 13 16:06:59 2022] PUT /identity/v3/projects/b48b1ff8f2904a37af937869f957a8c2/users/f6bfdc6bb6bf442cb43bb1d9b9093ed5/roles/15db4aaca3264ae29bde38903ba43471 => generated 0 bytes in 114 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 16:06:59.745656 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-85dcb61c-bc2b-4072-b16f-ea85df78a97e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:06:59.745806 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ef24157-c1ca-49d9-9e01-4677c5af61fe None None] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:06:59.746205 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ef24157-c1ca-49d9-9e01-4677c5af61fe None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:06:59.746567 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ef24157-c1ca-49d9-9e01-4677c5af61fe None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:06:59.822058 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.auth.core [None req-4ef24157-c1ca-49d9-9e01-4677c5af61fe None None] MFA Rules not processed for user `f6bfdc6bb6bf442cb43bb1d9b9093ed5`. Rule list: `[]` (Enabled: `True`). {{(pid=81437) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:06:59.823807 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-4ef24157-c1ca-49d9-9e01-4677c5af61fe None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:06:59.891853 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:06:59.914635 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 818/1663] 10.222.0.50 () {58 vars in 1018 bytes} [Tue Sep 13 16:06:59 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 224 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 13 16:06:59.977084 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85dcb61c-bc2b-4072-b16f-ea85df78a97e None None] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:06:59.978490 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-85dcb61c-bc2b-4072-b16f-ea85df78a97e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:07:00.004652 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85dcb61c-bc2b-4072-b16f-ea85df78a97e None None] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:07:00.015520 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-943542e4-9ce2-4891-bd47-40bfcc271f9c service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '31480b61418340bd88583f034f67412e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:07:00.016172 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-943542e4-9ce2-4891-bd47-40bfcc271f9c service placement] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:07:00.016675 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 846/1664] 10.222.0.50 () {58 vars in 958 bytes} [Tue Sep 13 16:06:59 2022] POST /identity/v3/auth/tokens => generated 3242 bytes in 272 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 13 16:07:00.019793 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-943542e4-9ce2-4891-bd47-40bfcc271f9c service placement] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:07:00.020147 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-943542e4-9ce2-4891-bd47-40bfcc271f9c service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:07:00.021035 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ef24157-c1ca-49d9-9e01-4677c5af61fe None None] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:07:00.048448 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-943542e4-9ce2-4891-bd47-40bfcc271f9c service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:07:00.049691 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-943542e4-9ce2-4891-bd47-40bfcc271f9c service placement] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:07:00.077493 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 819/1665] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 16:06:59 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 101 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 16:07:00.182495 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ef24157-c1ca-49d9-9e01-4677c5af61fe None None] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:07:00.195220 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bda6d952-e9fc-424d-ac1d-eb3a74950e21 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:07:00.196084 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bda6d952-e9fc-424d-ac1d-eb3a74950e21 admin admin] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:07:00.196457 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bda6d952-e9fc-424d-ac1d-eb3a74950e21 admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:07:00.197201 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bda6d952-e9fc-424d-ac1d-eb3a74950e21 admin admin] PATH_INFO: `/v3/projects` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:07:00.198879 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bda6d952-e9fc-424d-ac1d-eb3a74950e21 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:07:00.205431 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bda6d952-e9fc-424d-ac1d-eb3a74950e21 admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:07:00.233260 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:INSERT=3,keystone:DELETE=3,keystone:SELECT=94 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:07:00.275110 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 847/1666] 10.222.0.50 () {60 vars in 1153 bytes} [Tue Sep 13 16:07:00 2022] POST /identity/v3/projects => generated 363 bytes in 255 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 16:07:00.280423 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-943542e4-9ce2-4891-bd47-40bfcc271f9c service placement] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:07:00.304193 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-943542e4-9ce2-4891-bd47-40bfcc271f9c service placement] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:07:00.477900 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2511509b-78b6-46ce-9928-d56e199c5252 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:07:00.479022 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2511509b-78b6-46ce-9928-d56e199c5252 admin admin] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:07:00.479022 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2511509b-78b6-46ce-9928-d56e199c5252 admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:07:00.479204 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2511509b-78b6-46ce-9928-d56e199c5252 admin admin] PATH_INFO: `/v3/users` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:07:00.480024 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2511509b-78b6-46ce-9928-d56e199c5252 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:07:00.483473 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2511509b-78b6-46ce-9928-d56e199c5252 admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:07:00.504441 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bda6d952-e9fc-424d-ac1d-eb3a74950e21 admin admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:07:00.509584 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-bda6d952-e9fc-424d-ac1d-eb3a74950e21 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:07:00.539606 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 820/1667] 10.222.0.50 () {60 vars in 1144 bytes} [Tue Sep 13 16:07:00 2022] POST /identity/v3/users => generated 288 bytes in 261 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 16:07:00.541694 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bda6d952-e9fc-424d-ac1d-eb3a74950e21 admin admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:07:00.543722 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2511509b-78b6-46ce-9928-d56e199c5252 admin admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:07:00.565215 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b466aa5-0743-401e-8f3d-08cc6f9fa973 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:07:00.566788 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b466aa5-0743-401e-8f3d-08cc6f9fa973 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:07:00.569586 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b466aa5-0743-401e-8f3d-08cc6f9fa973 admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:07:00.578815 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b466aa5-0743-401e-8f3d-08cc6f9fa973 admin admin] PATH_INFO: `/v3/projects/936eda6390354d37b998e2268154fd6e` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:07:00.581100 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2511509b-78b6-46ce-9928-d56e199c5252 admin admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:07:00.583719 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b466aa5-0743-401e-8f3d-08cc6f9fa973 admin admin] RBAC: Authorizing `identity:delete_project(project_id=936eda6390354d37b998e2268154fd6e)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:07:00.586140 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b466aa5-0743-401e-8f3d-08cc6f9fa973 admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:07:00.596398 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88cab9a3-daec-499a-ab58-3a3d63e5745a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:07:00.599304 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88cab9a3-daec-499a-ab58-3a3d63e5745a admin admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:07:00.599585 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88cab9a3-daec-499a-ab58-3a3d63e5745a admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:07:00.599857 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88cab9a3-daec-499a-ab58-3a3d63e5745a admin admin] PATH_INFO: `/v3/roles` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:07:00.600516 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88cab9a3-daec-499a-ab58-3a3d63e5745a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:07:00.608819 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88cab9a3-daec-499a-ab58-3a3d63e5745a admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:07:00.648380 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 821/1668] 10.222.0.50 () {58 vars in 1122 bytes} [Tue Sep 13 16:07:00 2022] GET /identity/v3/roles => generated 1420 bytes in 106 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 16:07:00.648946 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88cab9a3-daec-499a-ab58-3a3d63e5745a admin admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:07:00.687558 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88cab9a3-daec-499a-ab58-3a3d63e5745a admin admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:07:00.698736 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-8b466aa5-0743-401e-8f3d-08cc6f9fa973 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '936eda6390354d37b998e2268154fd6e', 'request_id': 'req-8b466aa5-0743-401e-8f3d-08cc6f9fa973'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:07:00.725828 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-8b466aa5-0743-401e-8f3d-08cc6f9fa973 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '936eda6390354d37b998e2268154fd6e', 'request_id': 'req-8b466aa5-0743-401e-8f3d-08cc6f9fa973'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:07:00.737824 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-8b466aa5-0743-401e-8f3d-08cc6f9fa973 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '936eda6390354d37b998e2268154fd6e', 'request_id': 'req-8b466aa5-0743-401e-8f3d-08cc6f9fa973'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:07:00.764356 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-8b466aa5-0743-401e-8f3d-08cc6f9fa973 admin admin] The token cache is being invalidate because project 936eda6390354d37b998e2268154fd6e was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81437) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 13 16:07:00.765269 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-8b466aa5-0743-401e-8f3d-08cc6f9fa973 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:07:00.766399 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-8b466aa5-0743-401e-8f3d-08cc6f9fa973 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:07:00.773582 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 848/1669] 10.222.0.50 () {58 vars in 1233 bytes} [Tue Sep 13 16:07:00 2022] DELETE /identity/v3/projects/936eda6390354d37b998e2268154fd6e => generated 0 bytes in 271 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 16:07:00.781127 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9608f931-b29f-4237-b360-5e6ae539f883 None None] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:07:00.781434 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9608f931-b29f-4237-b360-5e6ae539f883 None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:07:00.781852 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9608f931-b29f-4237-b360-5e6ae539f883 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:07:00.824128 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.auth.core [None req-9608f931-b29f-4237-b360-5e6ae539f883 None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81437) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:07:00.825879 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-9608f931-b29f-4237-b360-5e6ae539f883 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:07:00.831760 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-118e05c3-a5c5-42cc-aebe-fbaa1191ad1a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:07:00.832513 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-118e05c3-a5c5-42cc-aebe-fbaa1191ad1a admin admin] REQUEST_METHOD: `PUT` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:07:00.832980 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-118e05c3-a5c5-42cc-aebe-fbaa1191ad1a admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:07:00.833393 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-118e05c3-a5c5-42cc-aebe-fbaa1191ad1a admin admin] PATH_INFO: `/v3/projects/499a8465673141aba3c5331ceaa17a82/users/f3d13f6b016648c7ab0d307612b229a6/roles/031d4e1a23e34f71858fc4a6dfbd4ccb` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:07:00.847658 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 849/1670] 10.222.0.50 () {58 vars in 958 bytes} [Tue Sep 13 16:07:00 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 68 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 13 16:07:00.856694 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb79a9ad-e361-4840-8cf1-1cdd0b527b1f None None] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:07:00.856976 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb79a9ad-e361-4840-8cf1-1cdd0b527b1f None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:07:00.857254 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb79a9ad-e361-4840-8cf1-1cdd0b527b1f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:07:00.874099 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-118e05c3-a5c5-42cc-aebe-fbaa1191ad1a admin admin] RBAC: Authorizing `identity:create_grant(project_id=499a8465673141aba3c5331ceaa17a82, user_id=f3d13f6b016648c7ab0d307612b229a6, role_id=031d4e1a23e34f71858fc4a6dfbd4ccb)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:07:00.876945 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-118e05c3-a5c5-42cc-aebe-fbaa1191ad1a admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:07:00.894265 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.auth.core [None req-cb79a9ad-e361-4840-8cf1-1cdd0b527b1f None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81437) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:07:00.900433 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 822/1671] 10.222.0.50 () {60 vars in 1483 bytes} [Tue Sep 13 16:07:00 2022] PUT /identity/v3/projects/499a8465673141aba3c5331ceaa17a82/users/f3d13f6b016648c7ab0d307612b229a6/roles/031d4e1a23e34f71858fc4a6dfbd4ccb => generated 0 bytes in 252 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 16:07:00.903914 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-cb79a9ad-e361-4840-8cf1-1cdd0b527b1f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:07:00.904605 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-118e05c3-a5c5-42cc-aebe-fbaa1191ad1a admin admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:07:00.906547 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-118e05c3-a5c5-42cc-aebe-fbaa1191ad1a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:07:01.002422 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:07:01.029976 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 850/1672] 10.222.0.50 () {58 vars in 958 bytes} [Tue Sep 13 16:07:00 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 174 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 13 16:07:01.034061 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb79a9ad-e361-4840-8cf1-1cdd0b527b1f None None] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:07:01.044798 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-118e05c3-a5c5-42cc-aebe-fbaa1191ad1a admin admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:07:01.053641 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb79a9ad-e361-4840-8cf1-1cdd0b527b1f None None] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:07:01.068799 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1dda922b-30a4-4343-9252-cc74aaf71012 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:07:01.069445 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dda922b-30a4-4343-9252-cc74aaf71012 admin admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:07:01.069795 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dda922b-30a4-4343-9252-cc74aaf71012 admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:07:01.070174 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dda922b-30a4-4343-9252-cc74aaf71012 admin admin] PATH_INFO: `/v3/domains` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:07:01.070848 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1dda922b-30a4-4343-9252-cc74aaf71012 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81437) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 16:07:01.071165 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1dda922b-30a4-4343-9252-cc74aaf71012 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:07:01.071856 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7285de9-6229-406c-9b6a-4d6f59256296 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:07:01.072632 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7285de9-6229-406c-9b6a-4d6f59256296 admin admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:07:01.072800 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1dda922b-30a4-4343-9252-cc74aaf71012 admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:07:01.073187 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7285de9-6229-406c-9b6a-4d6f59256296 admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:07:01.073589 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7285de9-6229-406c-9b6a-4d6f59256296 admin admin] PATH_INFO: `/v3/roles` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:07:01.074468 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7285de9-6229-406c-9b6a-4d6f59256296 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:07:01.077640 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7285de9-6229-406c-9b6a-4d6f59256296 admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:07:01.090649 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 823/1673] 10.222.0.50 () {58 vars in 1122 bytes} [Tue Sep 13 16:07:00 2022] GET /identity/v3/roles => generated 1420 bytes in 186 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 13 16:07:01.096152 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 851/1674] 10.222.0.50 () {58 vars in 1153 bytes} [Tue Sep 13 16:07:01 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 62 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 13 16:07:01.097719 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7285de9-6229-406c-9b6a-4d6f59256296 admin admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:07:01.102946 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1dda922b-30a4-4343-9252-cc74aaf71012 admin admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:07:01.133309 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7285de9-6229-406c-9b6a-4d6f59256296 admin admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:07:01.136152 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1dda922b-30a4-4343-9252-cc74aaf71012 admin admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:07:01.151601 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64c8bba7-3086-40e6-88a0-b8595ac70e3a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:07:01.152340 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64c8bba7-3086-40e6-88a0-b8595ac70e3a admin admin] REQUEST_METHOD: `PUT` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:07:01.152484 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64c8bba7-3086-40e6-88a0-b8595ac70e3a admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:07:01.152770 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64c8bba7-3086-40e6-88a0-b8595ac70e3a admin admin] PATH_INFO: `/v3/projects/499a8465673141aba3c5331ceaa17a82/users/f3d13f6b016648c7ab0d307612b229a6/roles/031d4e1a23e34f71858fc4a6dfbd4ccb` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:07:01.157617 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37d0cb54-9253-4d49-9cfa-62be1b620a39 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:07:01.158722 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37d0cb54-9253-4d49-9cfa-62be1b620a39 admin admin] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:07:01.159058 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37d0cb54-9253-4d49-9cfa-62be1b620a39 admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:07:01.159337 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37d0cb54-9253-4d49-9cfa-62be1b620a39 admin admin] PATH_INFO: `/v3/projects` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:07:01.159981 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37d0cb54-9253-4d49-9cfa-62be1b620a39 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:07:01.163090 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64c8bba7-3086-40e6-88a0-b8595ac70e3a admin admin] RBAC: Authorizing `identity:create_grant(project_id=499a8465673141aba3c5331ceaa17a82, user_id=f3d13f6b016648c7ab0d307612b229a6, role_id=031d4e1a23e34f71858fc4a6dfbd4ccb)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:07:01.167455 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37d0cb54-9253-4d49-9cfa-62be1b620a39 admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:07:01.171179 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64c8bba7-3086-40e6-88a0-b8595ac70e3a admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:07:01.196205 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 824/1675] 10.222.0.50 () {60 vars in 1483 bytes} [Tue Sep 13 16:07:01 2022] PUT /identity/v3/projects/499a8465673141aba3c5331ceaa17a82/users/f3d13f6b016648c7ab0d307612b229a6/roles/031d4e1a23e34f71858fc4a6dfbd4ccb => generated 0 bytes in 100 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 16:07:01.206816 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35c695aa-776d-4004-82ba-cfd3da325391 None None] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:07:01.207247 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35c695aa-776d-4004-82ba-cfd3da325391 None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:07:01.207593 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35c695aa-776d-4004-82ba-cfd3da325391 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:07:01.235693 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 852/1676] 10.222.0.50 () {60 vars in 1153 bytes} [Tue Sep 13 16:07:01 2022] POST /identity/v3/projects => generated 385 bytes in 130 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 16:07:01.236963 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37d0cb54-9253-4d49-9cfa-62be1b620a39 admin admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:07:01.264985 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.auth.core [None req-35c695aa-776d-4004-82ba-cfd3da325391 None None] MFA Rules not processed for user `f3d13f6b016648c7ab0d307612b229a6`. Rule list: `[]` (Enabled: `True`). {{(pid=81438) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:07:01.268463 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-35c695aa-776d-4004-82ba-cfd3da325391 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:07:01.313519 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=90,keystone:INSERT=3,keystone:DELETE=7 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:07:01.390933 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37d0cb54-9253-4d49-9cfa-62be1b620a39 admin admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:07:01.404166 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cd9d49d-318f-4507-8240-ca61cd2b2bc1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:07:01.405351 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cd9d49d-318f-4507-8240-ca61cd2b2bc1 admin admin] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:07:01.405839 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cd9d49d-318f-4507-8240-ca61cd2b2bc1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:07:01.406788 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cd9d49d-318f-4507-8240-ca61cd2b2bc1 admin admin] PATH_INFO: `/v3/users` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:07:01.408089 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3cd9d49d-318f-4507-8240-ca61cd2b2bc1 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:07:01.414301 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3cd9d49d-318f-4507-8240-ca61cd2b2bc1 admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:07:01.445540 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 825/1677] 10.222.0.50 () {58 vars in 958 bytes} [Tue Sep 13 16:07:01 2022] POST /identity/v3/auth/tokens => generated 3307 bytes in 240 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:07:01.458359 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35c695aa-776d-4004-82ba-cfd3da325391 None None] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:07:01.461040 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-35c695aa-776d-4004-82ba-cfd3da325391 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:07:01.464270 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 853/1678] 10.222.0.50 () {60 vars in 1144 bytes} [Tue Sep 13 16:07:01 2022] POST /identity/v3/users => generated 293 bytes in 228 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 16:07:01.466553 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cd9d49d-318f-4507-8240-ca61cd2b2bc1 admin admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:07:01.498830 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cd9d49d-318f-4507-8240-ca61cd2b2bc1 admin admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:07:01.517501 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59e523c7-bae8-45f0-bb27-0d72d0022f34 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:07:01.520669 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59e523c7-bae8-45f0-bb27-0d72d0022f34 admin admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:07:01.521153 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59e523c7-bae8-45f0-bb27-0d72d0022f34 admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:07:01.522630 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59e523c7-bae8-45f0-bb27-0d72d0022f34 admin admin] PATH_INFO: `/v3/roles` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:07:01.523988 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59e523c7-bae8-45f0-bb27-0d72d0022f34 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:07:01.526582 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59e523c7-bae8-45f0-bb27-0d72d0022f34 admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:07:01.556099 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 854/1679] 10.222.0.50 () {58 vars in 1122 bytes} [Tue Sep 13 16:07:01 2022] GET /identity/v3/roles => generated 1420 bytes in 89 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 16:07:01.559047 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59e523c7-bae8-45f0-bb27-0d72d0022f34 admin admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:07:01.590463 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59e523c7-bae8-45f0-bb27-0d72d0022f34 admin admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:07:01.605122 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0437b8fa-341d-4511-9e70-3088f5624a8b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:07:01.610372 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0437b8fa-341d-4511-9e70-3088f5624a8b admin admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:07:01.611573 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0437b8fa-341d-4511-9e70-3088f5624a8b admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:07:01.612297 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0437b8fa-341d-4511-9e70-3088f5624a8b admin admin] PATH_INFO: `/v3/roles` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:07:01.614200 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0437b8fa-341d-4511-9e70-3088f5624a8b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:07:01.620347 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0437b8fa-341d-4511-9e70-3088f5624a8b admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:07:01.638664 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35c695aa-776d-4004-82ba-cfd3da325391 None None] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:07:01.658039 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38709768-d9ed-4e12-9f8f-30013c80aa07 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7fd6b5afd67b48f9b0f0597ccd4cf4df', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:07:01.661300 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38709768-d9ed-4e12-9f8f-30013c80aa07 service cinder] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:07:01.661768 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38709768-d9ed-4e12-9f8f-30013c80aa07 service cinder] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:07:01.663437 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38709768-d9ed-4e12-9f8f-30013c80aa07 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:07:01.667255 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 855/1680] 10.222.0.50 () {58 vars in 1122 bytes} [Tue Sep 13 16:07:01 2022] GET /identity/v3/roles => generated 1420 bytes in 110 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 13 16:07:01.671865 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0437b8fa-341d-4511-9e70-3088f5624a8b admin admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:07:01.689066 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0437b8fa-341d-4511-9e70-3088f5624a8b admin admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:07:01.689733 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38709768-d9ed-4e12-9f8f-30013c80aa07 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:07:01.691682 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38709768-d9ed-4e12-9f8f-30013c80aa07 service cinder] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:07:01.696195 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b86bbf79-0764-43f1-9130-2935f491737d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:07:01.698101 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b86bbf79-0764-43f1-9130-2935f491737d admin admin] REQUEST_METHOD: `PUT` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:07:01.698607 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b86bbf79-0764-43f1-9130-2935f491737d admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:07:01.699631 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b86bbf79-0764-43f1-9130-2935f491737d admin admin] PATH_INFO: `/v3/projects/a76428a823684e04856476e54d140d61/users/9429939ab37a4d94856cb5cd2aba5e18/roles/15db4aaca3264ae29bde38903ba43471` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:07:01.721755 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 826/1681] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 16:07:01 2022] GET /identity/v3/auth/tokens => generated 3307 bytes in 264 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 13 16:07:01.733123 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b86bbf79-0764-43f1-9130-2935f491737d admin admin] RBAC: Authorizing `identity:create_grant(project_id=a76428a823684e04856476e54d140d61, user_id=9429939ab37a4d94856cb5cd2aba5e18, role_id=15db4aaca3264ae29bde38903ba43471)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:07:01.736933 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b86bbf79-0764-43f1-9130-2935f491737d admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:07:01.778074 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 856/1682] 10.222.0.50 () {60 vars in 1483 bytes} [Tue Sep 13 16:07:01 2022] PUT /identity/v3/projects/a76428a823684e04856476e54d140d61/users/9429939ab37a4d94856cb5cd2aba5e18/roles/15db4aaca3264ae29bde38903ba43471 => generated 0 bytes in 106 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 16:07:01.790843 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44a99c89-7ab3-41d1-a992-9e684bbb7967 None None] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:07:01.791326 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44a99c89-7ab3-41d1-a992-9e684bbb7967 None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:07:01.791674 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44a99c89-7ab3-41d1-a992-9e684bbb7967 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:07:01.892786 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.auth.core [None req-44a99c89-7ab3-41d1-a992-9e684bbb7967 None None] MFA Rules not processed for user `9429939ab37a4d94856cb5cd2aba5e18`. Rule list: `[]` (Enabled: `True`). {{(pid=81438) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:07:01.894114 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-44a99c89-7ab3-41d1-a992-9e684bbb7967 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:07:02.012154 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:07:02.048923 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b86bbf79-0764-43f1-9130-2935f491737d admin admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:07:02.052192 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-b86bbf79-0764-43f1-9130-2935f491737d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:07:02.069254 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 827/1683] 10.222.0.50 () {58 vars in 958 bytes} [Tue Sep 13 16:07:01 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 280 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:07:02.070256 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44a99c89-7ab3-41d1-a992-9e684bbb7967 None None] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:07:02.071812 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-44a99c89-7ab3-41d1-a992-9e684bbb7967 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:07:02.194535 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b86bbf79-0764-43f1-9130-2935f491737d admin admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:07:02.216218 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44ef9692-fa1d-44eb-9750-e92a6ea32b63 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7fd6b5afd67b48f9b0f0597ccd4cf4df', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:07:02.217808 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44ef9692-fa1d-44eb-9750-e92a6ea32b63 service cinder] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:07:02.218101 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44ef9692-fa1d-44eb-9750-e92a6ea32b63 service cinder] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:07:02.218843 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44ef9692-fa1d-44eb-9750-e92a6ea32b63 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:07:02.222870 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-44ef9692-fa1d-44eb-9750-e92a6ea32b63 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:07:02.230638 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44a99c89-7ab3-41d1-a992-9e684bbb7967 None None] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:07:02.244874 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db801bda-4531-48c9-8090-9934e096a56e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:07:02.246603 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db801bda-4531-48c9-8090-9934e096a56e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:07:02.248330 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db801bda-4531-48c9-8090-9934e096a56e admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:07:02.248931 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db801bda-4531-48c9-8090-9934e096a56e admin admin] PATH_INFO: `/v3/users/5d421769b0c84034b83f6336f71d72bf` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:07:02.251101 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db801bda-4531-48c9-8090-9934e096a56e admin admin] RBAC: Authorizing `identity:delete_user(user_id=5d421769b0c84034b83f6336f71d72bf)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:07:02.254359 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db801bda-4531-48c9-8090-9934e096a56e admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:07:02.431138 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-db801bda-4531-48c9-8090-9934e096a56e admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '5d421769b0c84034b83f6336f71d72bf'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:07:02.458642 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44ef9692-fa1d-44eb-9750-e92a6ea32b63 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:07:02.461345 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44ef9692-fa1d-44eb-9750-e92a6ea32b63 service cinder] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:07:02.461623 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-db801bda-4531-48c9-8090-9934e096a56e admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '5d421769b0c84034b83f6336f71d72bf'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:07:02.491403 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 857/1684] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 16:07:02 2022] GET /identity/v3/auth/tokens => generated 3242 bytes in 445 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 13 16:07:02.492002 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44ef9692-fa1d-44eb-9750-e92a6ea32b63 service cinder] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:07:02.506950 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-db801bda-4531-48c9-8090-9934e096a56e admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '5d421769b0c84034b83f6336f71d72bf'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:07:02.511068 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-db801bda-4531-48c9-8090-9934e096a56e admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '5d421769b0c84034b83f6336f71d72bf'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:07:02.520493 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44ef9692-fa1d-44eb-9750-e92a6ea32b63 service cinder] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:07:02.534695 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-db801bda-4531-48c9-8090-9934e096a56e admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '5d421769b0c84034b83f6336f71d72bf'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:07:02.542658 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b60d34e-3743-4bc3-a70e-f39d2a9e7014 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:07:02.543329 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b60d34e-3743-4bc3-a70e-f39d2a9e7014 admin admin] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:07:02.543704 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b60d34e-3743-4bc3-a70e-f39d2a9e7014 admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:07:02.544098 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b60d34e-3743-4bc3-a70e-f39d2a9e7014 admin admin] PATH_INFO: `/v3/projects` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:07:02.544628 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b60d34e-3743-4bc3-a70e-f39d2a9e7014 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:07:02.546707 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b60d34e-3743-4bc3-a70e-f39d2a9e7014 admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:07:02.550974 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 828/1685] 10.222.0.50 () {58 vars in 1224 bytes} [Tue Sep 13 16:07:02 2022] DELETE /identity/v3/users/5d421769b0c84034b83f6336f71d72bf => generated 0 bytes in 481 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 16:07:02.557142 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9af12b6-6a0b-410b-9c0e-f70fa17f6b0f None None] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:07:02.557430 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9af12b6-6a0b-410b-9c0e-f70fa17f6b0f None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:07:02.557698 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9af12b6-6a0b-410b-9c0e-f70fa17f6b0f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:07:02.623931 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 858/1686] 10.222.0.50 () {60 vars in 1153 bytes} [Tue Sep 13 16:07:02 2022] POST /identity/v3/projects => generated 387 bytes in 132 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 16:07:02.625049 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b60d34e-3743-4bc3-a70e-f39d2a9e7014 admin admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:07:02.631149 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-2b60d34e-3743-4bc3-a70e-f39d2a9e7014 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:07:02.636972 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.auth.core [None req-c9af12b6-6a0b-410b-9c0e-f70fa17f6b0f None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81438) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:07:02.642665 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-c9af12b6-6a0b-410b-9c0e-f70fa17f6b0f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:07:02.672919 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:07:02.844923 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b60d34e-3743-4bc3-a70e-f39d2a9e7014 admin admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:07:02.852046 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c5fdec7-da5e-409c-8306-bd2831d1438e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:07:02.852957 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c5fdec7-da5e-409c-8306-bd2831d1438e admin admin] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:07:02.853445 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c5fdec7-da5e-409c-8306-bd2831d1438e admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:07:02.853919 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c5fdec7-da5e-409c-8306-bd2831d1438e admin admin] PATH_INFO: `/v3/users` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:07:02.854777 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c5fdec7-da5e-409c-8306-bd2831d1438e admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:07:02.857154 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c5fdec7-da5e-409c-8306-bd2831d1438e admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:07:02.870294 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 829/1687] 10.222.0.50 () {58 vars in 958 bytes} [Tue Sep 13 16:07:02 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 311 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:07:02.890469 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 859/1688] 10.222.0.50 () {60 vars in 1144 bytes} [Tue Sep 13 16:07:02 2022] POST /identity/v3/users => generated 300 bytes in 266 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 16:07:02.894069 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9af12b6-6a0b-410b-9c0e-f70fa17f6b0f None None] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:07:02.915091 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9af12b6-6a0b-410b-9c0e-f70fa17f6b0f None None] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:07:02.926725 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79321db9-91e8-4641-88db-d5b26ff4eef7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:07:02.928763 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79321db9-91e8-4641-88db-d5b26ff4eef7 admin admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:07:02.929048 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79321db9-91e8-4641-88db-d5b26ff4eef7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:07:02.929321 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79321db9-91e8-4641-88db-d5b26ff4eef7 admin admin] PATH_INFO: `/v3/roles` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:07:02.929993 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79321db9-91e8-4641-88db-d5b26ff4eef7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:07:02.937326 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79321db9-91e8-4641-88db-d5b26ff4eef7 admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:07:02.943764 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:07:02.970605 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c5fdec7-da5e-409c-8306-bd2831d1438e admin admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:07:02.973201 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 830/1689] 10.222.0.50 () {58 vars in 1122 bytes} [Tue Sep 13 16:07:02 2022] GET /identity/v3/roles => generated 1420 bytes in 80 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 16:07:02.990226 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c5fdec7-da5e-409c-8306-bd2831d1438e admin admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:07:03.001465 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9c9d793-420e-4c72-89b0-648bce38fe67 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:07:03.003655 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9c9d793-420e-4c72-89b0-648bce38fe67 admin admin] REQUEST_METHOD: `PUT` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:07:03.005179 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9c9d793-420e-4c72-89b0-648bce38fe67 admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:07:03.005975 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9c9d793-420e-4c72-89b0-648bce38fe67 admin admin] PATH_INFO: `/v3/projects/d20c4bdf69f641a6958eb4cbdecbe9fa/users/83ae0a9676fd4be390cdaf0f5a2330d9/roles/031d4e1a23e34f71858fc4a6dfbd4ccb` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:07:03.039788 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79321db9-91e8-4641-88db-d5b26ff4eef7 admin admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:07:03.041893 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-79321db9-91e8-4641-88db-d5b26ff4eef7 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:07:03.051513 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9c9d793-420e-4c72-89b0-648bce38fe67 admin admin] RBAC: Authorizing `identity:create_grant(project_id=d20c4bdf69f641a6958eb4cbdecbe9fa, user_id=83ae0a9676fd4be390cdaf0f5a2330d9, role_id=031d4e1a23e34f71858fc4a6dfbd4ccb)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:07:03.056919 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9c9d793-420e-4c72-89b0-648bce38fe67 admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:07:03.090404 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 860/1690] 10.222.0.50 () {60 vars in 1483 bytes} [Tue Sep 13 16:07:02 2022] PUT /identity/v3/projects/d20c4bdf69f641a6958eb4cbdecbe9fa/users/83ae0a9676fd4be390cdaf0f5a2330d9/roles/031d4e1a23e34f71858fc4a6dfbd4ccb => generated 0 bytes in 118 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 16:07:03.093221 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9c9d793-420e-4c72-89b0-648bce38fe67 admin admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:07:03.166574 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79321db9-91e8-4641-88db-d5b26ff4eef7 admin admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:07:03.244710 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9c9d793-420e-4c72-89b0-648bce38fe67 admin admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:07:03.259232 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5e7d243-4bbd-470b-86e3-d8045a7a0863 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:07:03.259918 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5e7d243-4bbd-470b-86e3-d8045a7a0863 admin admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:07:03.260191 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5e7d243-4bbd-470b-86e3-d8045a7a0863 admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:07:03.260549 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5e7d243-4bbd-470b-86e3-d8045a7a0863 admin admin] PATH_INFO: `/v3/roles` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:07:03.261137 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5e7d243-4bbd-470b-86e3-d8045a7a0863 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:07:03.269456 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5e7d243-4bbd-470b-86e3-d8045a7a0863 admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:07:03.307135 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 861/1691] 10.222.0.50 () {58 vars in 1122 bytes} [Tue Sep 13 16:07:03 2022] GET /identity/v3/roles => generated 1420 bytes in 215 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 16:07:03.311702 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5e7d243-4bbd-470b-86e3-d8045a7a0863 admin admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:07:03.320589 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e86389b-db24-435c-9d1e-7fe94bdc9436 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f436f07939d4855a21b195bc7566244', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:07:03.321298 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e86389b-db24-435c-9d1e-7fe94bdc9436 service neutron] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:07:03.321494 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e86389b-db24-435c-9d1e-7fe94bdc9436 service neutron] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:07:03.321832 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e86389b-db24-435c-9d1e-7fe94bdc9436 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:07:03.342161 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5e7d243-4bbd-470b-86e3-d8045a7a0863 admin admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:07:03.352929 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-641c376b-3904-4313-9906-56ecf23ac0cd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:07:03.354671 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-641c376b-3904-4313-9906-56ecf23ac0cd admin admin] REQUEST_METHOD: `PUT` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:07:03.355534 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-641c376b-3904-4313-9906-56ecf23ac0cd admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:07:03.355918 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-641c376b-3904-4313-9906-56ecf23ac0cd admin admin] PATH_INFO: `/v3/projects/d20c4bdf69f641a6958eb4cbdecbe9fa/users/83ae0a9676fd4be390cdaf0f5a2330d9/roles/031d4e1a23e34f71858fc4a6dfbd4ccb` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:07:03.359143 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-641c376b-3904-4313-9906-56ecf23ac0cd admin admin] RBAC: Authorizing `identity:create_grant(project_id=d20c4bdf69f641a6958eb4cbdecbe9fa, user_id=83ae0a9676fd4be390cdaf0f5a2330d9, role_id=031d4e1a23e34f71858fc4a6dfbd4ccb)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:07:03.359837 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e86389b-db24-435c-9d1e-7fe94bdc9436 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:07:03.363665 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-641c376b-3904-4313-9906-56ecf23ac0cd admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:07:03.364699 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e86389b-db24-435c-9d1e-7fe94bdc9436 service neutron] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:07:03.383365 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 862/1692] 10.222.0.50 () {60 vars in 1483 bytes} [Tue Sep 13 16:07:03 2022] PUT /identity/v3/projects/d20c4bdf69f641a6958eb4cbdecbe9fa/users/83ae0a9676fd4be390cdaf0f5a2330d9/roles/031d4e1a23e34f71858fc4a6dfbd4ccb => generated 0 bytes in 72 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 16:07:03.394357 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a8cd340-e4cc-48d9-8cc0-ce549ebf63c0 None None] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:07:03.395476 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a8cd340-e4cc-48d9-8cc0-ce549ebf63c0 None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:07:03.396745 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a8cd340-e4cc-48d9-8cc0-ce549ebf63c0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:07:03.396921 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 831/1693] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 16:07:03 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 358 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 13 16:07:03.467420 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.auth.core [None req-8a8cd340-e4cc-48d9-8cc0-ce549ebf63c0 None None] MFA Rules not processed for user `83ae0a9676fd4be390cdaf0f5a2330d9`. Rule list: `[]` (Enabled: `True`). {{(pid=81437) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:07:03.468704 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-8a8cd340-e4cc-48d9-8cc0-ce549ebf63c0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:07:03.560557 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:07:03.604070 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 863/1694] 10.222.0.50 () {58 vars in 958 bytes} [Tue Sep 13 16:07:03 2022] POST /identity/v3/auth/tokens => generated 3331 bytes in 211 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:07:03.615065 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e86389b-db24-435c-9d1e-7fe94bdc9436 service neutron] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:07:03.617387 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-9e86389b-db24-435c-9d1e-7fe94bdc9436 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:07:03.736024 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e86389b-db24-435c-9d1e-7fe94bdc9436 service neutron] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:07:03.842348 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46ec5abe-0fb5-4064-a968-3d98b81801cd service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7fd6b5afd67b48f9b0f0597ccd4cf4df', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:07:03.845482 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46ec5abe-0fb5-4064-a968-3d98b81801cd service cinder] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:07:03.845782 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46ec5abe-0fb5-4064-a968-3d98b81801cd service cinder] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:07:03.846063 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46ec5abe-0fb5-4064-a968-3d98b81801cd service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:07:03.862883 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46ec5abe-0fb5-4064-a968-3d98b81801cd service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:07:03.865145 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46ec5abe-0fb5-4064-a968-3d98b81801cd service cinder] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:07:03.893120 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 832/1695] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 16:07:03 2022] GET /identity/v3/auth/tokens => generated 3331 bytes in 279 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 16:07:03.956396 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a8cd340-e4cc-48d9-8cc0-ce549ebf63c0 None None] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:07:04.096191 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46ec5abe-0fb5-4064-a968-3d98b81801cd service cinder] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:07:04.097521 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-46ec5abe-0fb5-4064-a968-3d98b81801cd service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:07:04.116336 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a8cd340-e4cc-48d9-8cc0-ce549ebf63c0 None None] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:07:04.123056 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46ec5abe-0fb5-4064-a968-3d98b81801cd service cinder] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:07:04.139939 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b44cc799-71bd-4cfd-8f21-b7b3b08e99c4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:07:04.140606 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b44cc799-71bd-4cfd-8f21-b7b3b08e99c4 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:07:04.140890 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b44cc799-71bd-4cfd-8f21-b7b3b08e99c4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:07:04.141159 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e82652d4-5f46-4534-b4c7-142cb080c47c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:07:04.141296 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b44cc799-71bd-4cfd-8f21-b7b3b08e99c4 admin admin] PATH_INFO: `/v3/users/aa034994d95a4b82bf85138fdb07f4b4` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:07:04.141828 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e82652d4-5f46-4534-b4c7-142cb080c47c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:07:04.142111 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e82652d4-5f46-4534-b4c7-142cb080c47c admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:07:04.142550 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e82652d4-5f46-4534-b4c7-142cb080c47c admin admin] PATH_INFO: `/v3/users/9429939ab37a4d94856cb5cd2aba5e18` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:07:04.143819 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b44cc799-71bd-4cfd-8f21-b7b3b08e99c4 admin admin] RBAC: Authorizing `identity:delete_user(user_id=aa034994d95a4b82bf85138fdb07f4b4)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:07:04.144663 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e82652d4-5f46-4534-b4c7-142cb080c47c admin admin] RBAC: Authorizing `identity:delete_user(user_id=9429939ab37a4d94856cb5cd2aba5e18)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:07:04.145774 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b44cc799-71bd-4cfd-8f21-b7b3b08e99c4 admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:07:04.150227 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e82652d4-5f46-4534-b4c7-142cb080c47c admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:07:04.174546 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:07:04.253237 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-b44cc799-71bd-4cfd-8f21-b7b3b08e99c4 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'aa034994d95a4b82bf85138fdb07f4b4'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:07:04.276422 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-b44cc799-71bd-4cfd-8f21-b7b3b08e99c4 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'aa034994d95a4b82bf85138fdb07f4b4'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:07:04.305441 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-e82652d4-5f46-4534-b4c7-142cb080c47c admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '9429939ab37a4d94856cb5cd2aba5e18'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:07:04.315078 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-b44cc799-71bd-4cfd-8f21-b7b3b08e99c4 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'aa034994d95a4b82bf85138fdb07f4b4'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:07:04.316546 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-b44cc799-71bd-4cfd-8f21-b7b3b08e99c4 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'aa034994d95a4b82bf85138fdb07f4b4'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:07:04.327868 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-e82652d4-5f46-4534-b4c7-142cb080c47c admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '9429939ab37a4d94856cb5cd2aba5e18'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:07:04.329287 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-b44cc799-71bd-4cfd-8f21-b7b3b08e99c4 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'aa034994d95a4b82bf85138fdb07f4b4'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:07:04.351872 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 833/1696] 10.222.0.50 () {58 vars in 1224 bytes} [Tue Sep 13 16:07:04 2022] DELETE /identity/v3/users/aa034994d95a4b82bf85138fdb07f4b4 => generated 0 bytes in 256 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 16:07:04.377914 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-e82652d4-5f46-4534-b4c7-142cb080c47c admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '9429939ab37a4d94856cb5cd2aba5e18'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:07:04.378688 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-e82652d4-5f46-4534-b4c7-142cb080c47c admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '9429939ab37a4d94856cb5cd2aba5e18'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:07:04.386830 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-e82652d4-5f46-4534-b4c7-142cb080c47c admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '9429939ab37a4d94856cb5cd2aba5e18'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:07:04.397940 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 864/1697] 10.222.0.50 () {58 vars in 1224 bytes} [Tue Sep 13 16:07:03 2022] DELETE /identity/v3/users/9429939ab37a4d94856cb5cd2aba5e18 => generated 0 bytes in 442 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 16:07:04.405102 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ac217c1-7b7c-40a4-a08f-02a42e70ae0c None None] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:07:04.405852 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ac217c1-7b7c-40a4-a08f-02a42e70ae0c None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:07:04.406365 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ac217c1-7b7c-40a4-a08f-02a42e70ae0c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:07:04.453016 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.auth.core [None req-0ac217c1-7b7c-40a4-a08f-02a42e70ae0c None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81438) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:07:04.454280 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-0ac217c1-7b7c-40a4-a08f-02a42e70ae0c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:07:04.603821 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 834/1698] 10.222.0.50 () {58 vars in 958 bytes} [Tue Sep 13 16:07:04 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 201 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:07:04.663798 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e82652d4-5f46-4534-b4c7-142cb080c47c admin admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:07:04.665525 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-e82652d4-5f46-4534-b4c7-142cb080c47c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:07:04.901890 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e82652d4-5f46-4534-b4c7-142cb080c47c admin admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:07:04.919866 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-526c7221-6944-4884-8007-430067eb18ed service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f436f07939d4855a21b195bc7566244', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:07:04.920588 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-526c7221-6944-4884-8007-430067eb18ed service neutron] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:07:04.923116 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-526c7221-6944-4884-8007-430067eb18ed service neutron] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:07:04.923472 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-526c7221-6944-4884-8007-430067eb18ed service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:07:04.971938 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-526c7221-6944-4884-8007-430067eb18ed service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:07:04.973419 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-526c7221-6944-4884-8007-430067eb18ed service neutron] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:07:05.015058 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 865/1699] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 16:07:04 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 352 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 16:07:05.422025 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa1a3149-1c32-4ee4-8a64-671a1491b90f None None] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:07:05.422986 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa1a3149-1c32-4ee4-8a64-671a1491b90f None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:07:05.424259 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa1a3149-1c32-4ee4-8a64-671a1491b90f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:07:05.513717 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.auth.core [None req-fa1a3149-1c32-4ee4-8a64-671a1491b90f None None] MFA Rules not processed for user `179e2ae27b924db0a49b64502548525a`. Rule list: `[]` (Enabled: `True`). {{(pid=81438) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:07:05.516132 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-fa1a3149-1c32-4ee4-8a64-671a1491b90f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:07:05.740968 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 835/1700] 10.222.0.50 () {58 vars in 958 bytes} [Tue Sep 13 16:07:05 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 315 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:07:05.742782 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-808814c8-6f73-4552-bf3c-d79f41acacce None None] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:07:05.743345 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-808814c8-6f73-4552-bf3c-d79f41acacce None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:07:05.743415 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-808814c8-6f73-4552-bf3c-d79f41acacce None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:07:05.753327 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:07:05.771375 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa1a3149-1c32-4ee4-8a64-671a1491b90f None None] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:07:05.772826 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-fa1a3149-1c32-4ee4-8a64-671a1491b90f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:07:05.803561 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa1a3149-1c32-4ee4-8a64-671a1491b90f None None] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:07:05.806983 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.auth.core [None req-808814c8-6f73-4552-bf3c-d79f41acacce None None] MFA Rules not processed for user `179e2ae27b924db0a49b64502548525a`. Rule list: `[]` (Enabled: `True`). {{(pid=81437) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:07:05.808356 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-808814c8-6f73-4552-bf3c-d79f41acacce None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:07:05.812968 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9dc463c4-3d38-452b-b33b-b4629ae929b1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:07:05.813752 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dc463c4-3d38-452b-b33b-b4629ae929b1 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:07:05.814190 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dc463c4-3d38-452b-b33b-b4629ae929b1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:07:05.817564 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dc463c4-3d38-452b-b33b-b4629ae929b1 admin admin] PATH_INFO: `/v3/projects/e907c12bca4b4205a4877e3c6596db16` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:07:05.818943 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 866/1701] 10.222.0.50 () {58 vars in 958 bytes} [Tue Sep 13 16:07:05 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 77 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:07:05.820343 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5425612f-5919-4a16-8507-f7e7157fccee None None] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:07:05.820632 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5425612f-5919-4a16-8507-f7e7157fccee None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:07:05.820970 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5425612f-5919-4a16-8507-f7e7157fccee None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:07:05.825316 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9dc463c4-3d38-452b-b33b-b4629ae929b1 admin admin] RBAC: Authorizing `identity:delete_project(project_id=e907c12bca4b4205a4877e3c6596db16)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:07:05.834387 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: WARNING py.warnings [None req-9dc463c4-3d38-452b-b33b-b4629ae929b1 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:delete_project": "(role:admin and system_scope:all) or (role:admin and domain_id:%(target.project.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Sep 13 16:07:05.834387 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: warnings.warn(msg) Sep 13 16:07:05.834387 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: Sep 13 16:07:05.863197 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9dc463c4-3d38-452b-b33b-b4629ae929b1 admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:07:05.896060 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:DELETE=10,keystone:SELECT=89,keystone:INSERT=1 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:07:05.911529 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.auth.core [None req-5425612f-5919-4a16-8507-f7e7157fccee None None] MFA Rules not processed for user `e1cbafa1d3c244c5906e475494b39d4c`. Rule list: `[]` (Enabled: `True`). {{(pid=81437) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:07:05.915695 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-5425612f-5919-4a16-8507-f7e7157fccee None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:07:06.005760 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-9dc463c4-3d38-452b-b33b-b4629ae929b1 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'e907c12bca4b4205a4877e3c6596db16', 'request_id': 'req-9dc463c4-3d38-452b-b33b-b4629ae929b1'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:07:06.034591 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-9dc463c4-3d38-452b-b33b-b4629ae929b1 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'e907c12bca4b4205a4877e3c6596db16', 'request_id': 'req-9dc463c4-3d38-452b-b33b-b4629ae929b1'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:07:06.051937 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-9dc463c4-3d38-452b-b33b-b4629ae929b1 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'e907c12bca4b4205a4877e3c6596db16', 'request_id': 'req-9dc463c4-3d38-452b-b33b-b4629ae929b1'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:07:06.069858 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 867/1702] 10.222.0.50 () {58 vars in 958 bytes} [Tue Sep 13 16:07:05 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 250 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:07:06.071226 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5425612f-5919-4a16-8507-f7e7157fccee None None] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:07:06.073427 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-5425612f-5919-4a16-8507-f7e7157fccee None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:07:06.118611 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-9dc463c4-3d38-452b-b33b-b4629ae929b1 admin admin] The token cache is being invalidate because project e907c12bca4b4205a4877e3c6596db16 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81438) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 13 16:07:06.119864 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-9dc463c4-3d38-452b-b33b-b4629ae929b1 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:07:06.122442 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-9dc463c4-3d38-452b-b33b-b4629ae929b1 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:07:06.127027 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 836/1703] 10.222.0.50 () {58 vars in 1233 bytes} [Tue Sep 13 16:07:05 2022] DELETE /identity/v3/projects/e907c12bca4b4205a4877e3c6596db16 => generated 0 bytes in 356 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 16:07:06.128971 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9dc463c4-3d38-452b-b33b-b4629ae929b1 admin admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:07:06.134009 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-9dc463c4-3d38-452b-b33b-b4629ae929b1 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:07:06.316455 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5425612f-5919-4a16-8507-f7e7157fccee None None] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:07:06.343663 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9dc463c4-3d38-452b-b33b-b4629ae929b1 admin admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:07:06.358017 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20090fc4-98aa-49b0-b119-e45037a5fa89 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:07:06.358918 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20090fc4-98aa-49b0-b119-e45037a5fa89 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:07:06.359362 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20090fc4-98aa-49b0-b119-e45037a5fa89 admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:07:06.359781 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20090fc4-98aa-49b0-b119-e45037a5fa89 admin admin] PATH_INFO: `/v3/projects/d57e69de18ac46399beb36eaca6bf06a` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:07:06.361402 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20090fc4-98aa-49b0-b119-e45037a5fa89 admin admin] RBAC: Authorizing `identity:delete_project(project_id=d57e69de18ac46399beb36eaca6bf06a)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:07:06.364364 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20090fc4-98aa-49b0-b119-e45037a5fa89 admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:07:06.458695 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-20090fc4-98aa-49b0-b119-e45037a5fa89 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'd57e69de18ac46399beb36eaca6bf06a', 'request_id': 'req-20090fc4-98aa-49b0-b119-e45037a5fa89'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:07:06.474588 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-20090fc4-98aa-49b0-b119-e45037a5fa89 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'd57e69de18ac46399beb36eaca6bf06a', 'request_id': 'req-20090fc4-98aa-49b0-b119-e45037a5fa89'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:07:06.490551 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-20090fc4-98aa-49b0-b119-e45037a5fa89 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'd57e69de18ac46399beb36eaca6bf06a', 'request_id': 'req-20090fc4-98aa-49b0-b119-e45037a5fa89'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:07:06.515776 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-20090fc4-98aa-49b0-b119-e45037a5fa89 admin admin] The token cache is being invalidate because project d57e69de18ac46399beb36eaca6bf06a was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81438) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 13 16:07:06.516453 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-20090fc4-98aa-49b0-b119-e45037a5fa89 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:07:06.517286 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-20090fc4-98aa-49b0-b119-e45037a5fa89 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:07:06.520132 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 837/1704] 10.222.0.50 () {58 vars in 1233 bytes} [Tue Sep 13 16:07:06 2022] DELETE /identity/v3/projects/d57e69de18ac46399beb36eaca6bf06a => generated 0 bytes in 392 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 16:07:06.521446 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20090fc4-98aa-49b0-b119-e45037a5fa89 admin admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:07:06.523201 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-20090fc4-98aa-49b0-b119-e45037a5fa89 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:07:06.582130 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1dacbc6c-bfe4-4965-a54a-014bb20c3fb3 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4e039b46c5e3445bbb39904b00c5430e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'admin', 'ResellerAdmin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:07:06.584408 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dacbc6c-bfe4-4965-a54a-014bb20c3fb3 service nova] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:07:06.584694 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dacbc6c-bfe4-4965-a54a-014bb20c3fb3 service nova] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:07:06.584968 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dacbc6c-bfe4-4965-a54a-014bb20c3fb3 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:07:06.601801 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-1dacbc6c-bfe4-4965-a54a-014bb20c3fb3 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:07:06.615753 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=84,keystone:DELETE=14,keystone:INSERT=2 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:07:06.696077 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:07:06.703592 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20090fc4-98aa-49b0-b119-e45037a5fa89 admin admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:07:06.733365 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34088c62-8d1b-401e-a749-9db76f449dc1 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4e039b46c5e3445bbb39904b00c5430e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'admin', 'ResellerAdmin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:07:06.733949 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34088c62-8d1b-401e-a749-9db76f449dc1 service nova] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:07:06.734229 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34088c62-8d1b-401e-a749-9db76f449dc1 service nova] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:07:06.734504 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34088c62-8d1b-401e-a749-9db76f449dc1 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:07:06.736493 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-34088c62-8d1b-401e-a749-9db76f449dc1 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:07:06.805035 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1dacbc6c-bfe4-4965-a54a-014bb20c3fb3 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:07:06.807580 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1dacbc6c-bfe4-4965-a54a-014bb20c3fb3 service nova] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:07:06.846609 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 868/1705] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 16:07:06 2022] GET /identity/v3/auth/tokens => generated 3256 bytes in 776 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 16:07:06.848796 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7567eea2-3728-4646-a05a-86a7c0c98cd9 None None] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:07:06.850120 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7567eea2-3728-4646-a05a-86a7c0c98cd9 None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:07:06.850556 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7567eea2-3728-4646-a05a-86a7c0c98cd9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:07:06.940230 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.auth.core [None req-7567eea2-3728-4646-a05a-86a7c0c98cd9 None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81437) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:07:06.941564 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-7567eea2-3728-4646-a05a-86a7c0c98cd9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:07:06.948638 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34088c62-8d1b-401e-a749-9db76f449dc1 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:07:06.950755 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34088c62-8d1b-401e-a749-9db76f449dc1 service nova] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:07:06.990260 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 838/1706] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 16:07:06 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 469 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 16:07:07.013975 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34088c62-8d1b-401e-a749-9db76f449dc1 service nova] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:07:07.017414 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-34088c62-8d1b-401e-a749-9db76f449dc1 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:07:07.156785 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 869/1707] 10.222.0.50 () {58 vars in 958 bytes} [Tue Sep 13 16:07:06 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 309 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:07:07.159706 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7567eea2-3728-4646-a05a-86a7c0c98cd9 None None] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:07:07.168783 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-7567eea2-3728-4646-a05a-86a7c0c98cd9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:07:07.230390 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7567eea2-3728-4646-a05a-86a7c0c98cd9 None None] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:07:07.243760 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97e9e45b-fada-4bcc-b20a-226293367ec9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:07:07.244736 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97e9e45b-fada-4bcc-b20a-226293367ec9 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:07:07.245023 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97e9e45b-fada-4bcc-b20a-226293367ec9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:07:07.245445 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97e9e45b-fada-4bcc-b20a-226293367ec9 admin admin] PATH_INFO: `/v3/users/35aa294e5c9944de9550bf9ce05698fb` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:07:07.249597 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97e9e45b-fada-4bcc-b20a-226293367ec9 admin admin] RBAC: Authorizing `identity:delete_user(user_id=35aa294e5c9944de9550bf9ce05698fb)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:07:07.253928 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97e9e45b-fada-4bcc-b20a-226293367ec9 admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:07:07.258139 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34088c62-8d1b-401e-a749-9db76f449dc1 service nova] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:07:07.270089 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7010d1b-4e8e-4727-a475-402a3371857b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:07:07.270770 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7010d1b-4e8e-4727-a475-402a3371857b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:07:07.271130 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7010d1b-4e8e-4727-a475-402a3371857b admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:07:07.271324 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7010d1b-4e8e-4727-a475-402a3371857b admin admin] PATH_INFO: `/v3/projects/a76428a823684e04856476e54d140d61` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:07:07.278280 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7010d1b-4e8e-4727-a475-402a3371857b admin admin] RBAC: Authorizing `identity:delete_project(project_id=a76428a823684e04856476e54d140d61)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:07:07.282690 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7010d1b-4e8e-4727-a475-402a3371857b admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:07:07.425310 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-a7010d1b-4e8e-4727-a475-402a3371857b admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'a76428a823684e04856476e54d140d61', 'request_id': 'req-a7010d1b-4e8e-4727-a475-402a3371857b'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:07:07.435833 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-97e9e45b-fada-4bcc-b20a-226293367ec9 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '35aa294e5c9944de9550bf9ce05698fb'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:07:07.447494 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-a7010d1b-4e8e-4727-a475-402a3371857b admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'a76428a823684e04856476e54d140d61', 'request_id': 'req-a7010d1b-4e8e-4727-a475-402a3371857b'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:07:07.456090 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-97e9e45b-fada-4bcc-b20a-226293367ec9 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '35aa294e5c9944de9550bf9ce05698fb'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:07:07.474880 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-a7010d1b-4e8e-4727-a475-402a3371857b admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'a76428a823684e04856476e54d140d61', 'request_id': 'req-a7010d1b-4e8e-4727-a475-402a3371857b'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:07:07.493879 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-97e9e45b-fada-4bcc-b20a-226293367ec9 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '35aa294e5c9944de9550bf9ce05698fb'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:07:07.496887 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:07:07.497461 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-97e9e45b-fada-4bcc-b20a-226293367ec9 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '35aa294e5c9944de9550bf9ce05698fb'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:07:07.512708 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-a7010d1b-4e8e-4727-a475-402a3371857b admin admin] The token cache is being invalidate because project a76428a823684e04856476e54d140d61 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81438) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 13 16:07:07.512708 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-a7010d1b-4e8e-4727-a475-402a3371857b admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:07:07.517874 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-97e9e45b-fada-4bcc-b20a-226293367ec9 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '35aa294e5c9944de9550bf9ce05698fb'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:07:07.518759 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-a7010d1b-4e8e-4727-a475-402a3371857b admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:07:07.534570 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 839/1708] 10.222.0.50 () {58 vars in 1233 bytes} [Tue Sep 13 16:07:06 2022] DELETE /identity/v3/projects/a76428a823684e04856476e54d140d61 => generated 0 bytes in 538 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 16:07:07.534880 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 870/1709] 10.222.0.50 () {58 vars in 1224 bytes} [Tue Sep 13 16:07:07 2022] DELETE /identity/v3/users/35aa294e5c9944de9550bf9ce05698fb => generated 0 bytes in 372 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 16:07:07.534880 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13066b46-3e62-4461-84a2-34b51de82d2b None None] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:07:07.534880 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13066b46-3e62-4461-84a2-34b51de82d2b None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:07:07.534880 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13066b46-3e62-4461-84a2-34b51de82d2b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:07:07.536090 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c477e61-1d95-475b-9373-908615d24cf2 None None] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:07:07.536408 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c477e61-1d95-475b-9373-908615d24cf2 None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:07:07.536686 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c477e61-1d95-475b-9373-908615d24cf2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:07:07.612241 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.auth.core [None req-4c477e61-1d95-475b-9373-908615d24cf2 None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81438) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:07:07.613504 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-4c477e61-1d95-475b-9373-908615d24cf2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:07:07.621085 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.auth.core [None req-13066b46-3e62-4461-84a2-34b51de82d2b None None] MFA Rules not processed for user `c0fd4bbe430b4a8888f3be4b23982c5a`. Rule list: `[]` (Enabled: `True`). {{(pid=81437) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:07:07.624828 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-13066b46-3e62-4461-84a2-34b51de82d2b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:07:07.661700 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:07:07.782552 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 840/1710] 10.222.0.50 () {58 vars in 958 bytes} [Tue Sep 13 16:07:07 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 253 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:07:07.788384 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c477e61-1d95-475b-9373-908615d24cf2 None None] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:07:07.790876 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-4c477e61-1d95-475b-9373-908615d24cf2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:07:07.798509 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 871/1711] 10.222.0.50 () {58 vars in 958 bytes} [Tue Sep 13 16:07:07 2022] POST /identity/v3/auth/tokens => generated 3266 bytes in 268 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:07:07.799421 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13066b46-3e62-4461-84a2-34b51de82d2b None None] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:07:07.832249 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13066b46-3e62-4461-84a2-34b51de82d2b None None] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:07:07.842574 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c477e61-1d95-475b-9373-908615d24cf2 None None] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:07:07.844532 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11ad4454-74af-4154-b8ab-7d579c18b52a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:07:07.845157 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11ad4454-74af-4154-b8ab-7d579c18b52a admin admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:07:07.845433 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11ad4454-74af-4154-b8ab-7d579c18b52a admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:07:07.845760 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11ad4454-74af-4154-b8ab-7d579c18b52a admin admin] PATH_INFO: `/v3/domains` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:07:07.846743 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11ad4454-74af-4154-b8ab-7d579c18b52a admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81437) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 16:07:07.846743 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11ad4454-74af-4154-b8ab-7d579c18b52a admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:07:07.850155 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11ad4454-74af-4154-b8ab-7d579c18b52a admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:07:07.856032 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62fc2f6e-dacc-4e17-9708-ea92f71c2607 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:07:07.857812 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62fc2f6e-dacc-4e17-9708-ea92f71c2607 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:07:07.858370 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62fc2f6e-dacc-4e17-9708-ea92f71c2607 admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:07:07.858762 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62fc2f6e-dacc-4e17-9708-ea92f71c2607 admin admin] PATH_INFO: `/v3/users/83ae0a9676fd4be390cdaf0f5a2330d9` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:07:07.862107 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62fc2f6e-dacc-4e17-9708-ea92f71c2607 admin admin] RBAC: Authorizing `identity:delete_user(user_id=83ae0a9676fd4be390cdaf0f5a2330d9)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:07:07.867305 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62fc2f6e-dacc-4e17-9708-ea92f71c2607 admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:07:07.889623 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 872/1712] 10.222.0.50 () {58 vars in 1153 bytes} [Tue Sep 13 16:07:07 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 90 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 16:07:07.891001 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11ad4454-74af-4154-b8ab-7d579c18b52a admin admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:07:07.897759 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-11ad4454-74af-4154-b8ab-7d579c18b52a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:07:08.011577 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-62fc2f6e-dacc-4e17-9708-ea92f71c2607 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '83ae0a9676fd4be390cdaf0f5a2330d9'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:07:08.030764 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-62fc2f6e-dacc-4e17-9708-ea92f71c2607 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '83ae0a9676fd4be390cdaf0f5a2330d9'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:07:08.059264 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-62fc2f6e-dacc-4e17-9708-ea92f71c2607 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '83ae0a9676fd4be390cdaf0f5a2330d9'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:07:08.059773 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-62fc2f6e-dacc-4e17-9708-ea92f71c2607 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '83ae0a9676fd4be390cdaf0f5a2330d9'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:07:08.074033 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-62fc2f6e-dacc-4e17-9708-ea92f71c2607 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '83ae0a9676fd4be390cdaf0f5a2330d9'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:07:08.095563 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 841/1713] 10.222.0.50 () {58 vars in 1224 bytes} [Tue Sep 13 16:07:07 2022] DELETE /identity/v3/users/83ae0a9676fd4be390cdaf0f5a2330d9 => generated 0 bytes in 310 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 16:07:08.095563 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62fc2f6e-dacc-4e17-9708-ea92f71c2607 admin admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:07:08.098981 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-62fc2f6e-dacc-4e17-9708-ea92f71c2607 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:07:08.109530 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11ad4454-74af-4154-b8ab-7d579c18b52a admin admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:07:08.266727 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62fc2f6e-dacc-4e17-9708-ea92f71c2607 admin admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:07:08.274007 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06bd584e-b26a-41b7-8f4a-6f50d0621860 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4e039b46c5e3445bbb39904b00c5430e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'admin', 'ResellerAdmin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:07:08.275023 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06bd584e-b26a-41b7-8f4a-6f50d0621860 service nova] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:07:08.275302 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06bd584e-b26a-41b7-8f4a-6f50d0621860 service nova] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:07:08.275575 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06bd584e-b26a-41b7-8f4a-6f50d0621860 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:07:08.280124 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-06bd584e-b26a-41b7-8f4a-6f50d0621860 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:07:08.282672 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97a77d9d-e4aa-4d54-a041-67ad07f6a7ca admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:07:08.284446 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97a77d9d-e4aa-4d54-a041-67ad07f6a7ca admin admin] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:07:08.284865 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97a77d9d-e4aa-4d54-a041-67ad07f6a7ca admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:07:08.285361 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97a77d9d-e4aa-4d54-a041-67ad07f6a7ca admin admin] PATH_INFO: `/v3/projects` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:07:08.286172 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97a77d9d-e4aa-4d54-a041-67ad07f6a7ca admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:07:08.288468 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97a77d9d-e4aa-4d54-a041-67ad07f6a7ca admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:07:08.327790 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=88,keystone:DELETE=10,keystone:INSERT=2 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:07:08.362930 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 842/1714] 10.222.0.50 () {60 vars in 1153 bytes} [Tue Sep 13 16:07:08 2022] POST /identity/v3/projects => generated 371 bytes in 269 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 16:07:08.364808 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8635e2d-32e2-4854-b1d6-9d46e52e4a17 None None] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:07:08.365094 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8635e2d-32e2-4854-b1d6-9d46e52e4a17 None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:07:08.365385 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8635e2d-32e2-4854-b1d6-9d46e52e4a17 None None] PATH_INFO: `/` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:07:08.375560 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 843/1715] 10.222.0.50 () {54 vars in 912 bytes} [Tue Sep 13 16:07:08 2022] GET /identity => generated 270 bytes in 12 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 16:07:08.376556 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8635e2d-32e2-4854-b1d6-9d46e52e4a17 None None] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:07:08.384663 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:07:08.417384 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06bd584e-b26a-41b7-8f4a-6f50d0621860 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:07:08.424675 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06bd584e-b26a-41b7-8f4a-6f50d0621860 service nova] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:07:08.458639 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 873/1716] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 16:07:07 2022] GET /identity/v3/auth/tokens => generated 3266 bytes in 568 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 16:07:08.462621 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fa5faf9-74f5-43a4-847c-26b612b61c71 None None] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:07:08.462930 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fa5faf9-74f5-43a4-847c-26b612b61c71 None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:07:08.463225 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fa5faf9-74f5-43a4-847c-26b612b61c71 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:07:08.521847 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8635e2d-32e2-4854-b1d6-9d46e52e4a17 None None] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:07:08.557974 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39dd96f4-6a6d-4232-b1cd-3ef436738cba admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:07:08.560065 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39dd96f4-6a6d-4232-b1cd-3ef436738cba admin admin] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:07:08.562382 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39dd96f4-6a6d-4232-b1cd-3ef436738cba admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:07:08.563043 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39dd96f4-6a6d-4232-b1cd-3ef436738cba admin admin] PATH_INFO: `/v3/users` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:07:08.563904 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39dd96f4-6a6d-4232-b1cd-3ef436738cba admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:07:08.569667 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39dd96f4-6a6d-4232-b1cd-3ef436738cba admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:07:08.578222 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.auth.core [None req-4fa5faf9-74f5-43a4-847c-26b612b61c71 None None] MFA Rules not processed for user `2f436f07939d4855a21b195bc7566244`. Rule list: `[]` (Enabled: `True`). {{(pid=81437) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:07:08.579855 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-4fa5faf9-74f5-43a4-847c-26b612b61c71 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:07:08.647275 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 844/1717] 10.222.0.50 () {60 vars in 1144 bytes} [Tue Sep 13 16:07:08 2022] POST /identity/v3/users => generated 286 bytes in 271 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 16:07:08.653153 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39dd96f4-6a6d-4232-b1cd-3ef436738cba admin admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:07:08.670038 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 874/1718] 10.222.0.50 () {58 vars in 1009 bytes} [Tue Sep 13 16:07:08 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 211 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:07:08.687143 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39dd96f4-6a6d-4232-b1cd-3ef436738cba admin admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:07:08.700542 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c7bb72d-1dfd-4938-80f3-c650ad308ecf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:07:08.702053 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c7bb72d-1dfd-4938-80f3-c650ad308ecf admin admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:07:08.702444 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c7bb72d-1dfd-4938-80f3-c650ad308ecf admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:07:08.702820 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c7bb72d-1dfd-4938-80f3-c650ad308ecf admin admin] PATH_INFO: `/v3/roles` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:07:08.703611 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c7bb72d-1dfd-4938-80f3-c650ad308ecf admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:07:08.706737 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c7bb72d-1dfd-4938-80f3-c650ad308ecf admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:07:08.735543 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fa5faf9-74f5-43a4-847c-26b612b61c71 None None] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:07:08.737387 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-4fa5faf9-74f5-43a4-847c-26b612b61c71 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:07:08.765896 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fa5faf9-74f5-43a4-847c-26b612b61c71 None None] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:07:08.782185 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 845/1719] 10.222.0.50 () {58 vars in 1122 bytes} [Tue Sep 13 16:07:08 2022] GET /identity/v3/roles => generated 1420 bytes in 131 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 16:07:08.783855 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c7bb72d-1dfd-4938-80f3-c650ad308ecf admin admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:07:08.785094 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15ccd696-e5e8-4477-8bc8-03bea90239cd service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f436f07939d4855a21b195bc7566244', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:07:08.786711 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15ccd696-e5e8-4477-8bc8-03bea90239cd service neutron] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:07:08.788505 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15ccd696-e5e8-4477-8bc8-03bea90239cd service neutron] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:07:08.788959 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15ccd696-e5e8-4477-8bc8-03bea90239cd service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:07:08.813331 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15ccd696-e5e8-4477-8bc8-03bea90239cd service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:07:08.816827 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c7bb72d-1dfd-4938-80f3-c650ad308ecf admin admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:07:08.817573 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15ccd696-e5e8-4477-8bc8-03bea90239cd service neutron] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:07:08.826626 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59a2e50a-bcc0-427b-8290-caac4993729f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:07:08.828688 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59a2e50a-bcc0-427b-8290-caac4993729f admin admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:07:08.828818 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59a2e50a-bcc0-427b-8290-caac4993729f admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:07:08.829015 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59a2e50a-bcc0-427b-8290-caac4993729f admin admin] PATH_INFO: `/v3/roles` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:07:08.831652 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59a2e50a-bcc0-427b-8290-caac4993729f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:07:08.835698 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59a2e50a-bcc0-427b-8290-caac4993729f admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:07:08.838551 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 875/1720] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 16:07:08 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 104 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 13 16:07:08.857639 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15ccd696-e5e8-4477-8bc8-03bea90239cd service neutron] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:07:08.881816 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 846/1721] 10.222.0.50 () {58 vars in 1122 bytes} [Tue Sep 13 16:07:08 2022] GET /identity/v3/roles => generated 1420 bytes in 99 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 16:07:08.886871 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15ccd696-e5e8-4477-8bc8-03bea90239cd service neutron] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:07:08.900811 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f715fa75-dbf4-4f64-a67c-db34504a9ed1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:07:08.901497 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f715fa75-dbf4-4f64-a67c-db34504a9ed1 admin admin] REQUEST_METHOD: `PUT` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:07:08.901777 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f715fa75-dbf4-4f64-a67c-db34504a9ed1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:07:08.903400 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f715fa75-dbf4-4f64-a67c-db34504a9ed1 admin admin] PATH_INFO: `/v3/projects/87c0bc925183409993d04ed92503420f/users/11dc0707bcf84e3784afbb0031ab90ca/roles/15db4aaca3264ae29bde38903ba43471` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:07:08.938724 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f715fa75-dbf4-4f64-a67c-db34504a9ed1 admin admin] RBAC: Authorizing `identity:create_grant(project_id=87c0bc925183409993d04ed92503420f, user_id=11dc0707bcf84e3784afbb0031ab90ca, role_id=15db4aaca3264ae29bde38903ba43471)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:07:08.946735 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f715fa75-dbf4-4f64-a67c-db34504a9ed1 admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:07:08.955464 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59a2e50a-bcc0-427b-8290-caac4993729f admin admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:07:08.957796 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-59a2e50a-bcc0-427b-8290-caac4993729f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:07:09.003617 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 876/1722] 10.222.0.50 () {60 vars in 1483 bytes} [Tue Sep 13 16:07:08 2022] PUT /identity/v3/projects/87c0bc925183409993d04ed92503420f/users/11dc0707bcf84e3784afbb0031ab90ca/roles/15db4aaca3264ae29bde38903ba43471 => generated 0 bytes in 147 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 16:07:09.006344 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59a2e50a-bcc0-427b-8290-caac4993729f admin admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:07:09.018873 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faed3037-717d-448f-aeed-a5556a9e377f None None] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:07:09.019716 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faed3037-717d-448f-aeed-a5556a9e377f None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:07:09.020191 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faed3037-717d-448f-aeed-a5556a9e377f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:07:09.136053 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.auth.core [None req-faed3037-717d-448f-aeed-a5556a9e377f None None] MFA Rules not processed for user `11dc0707bcf84e3784afbb0031ab90ca`. Rule list: `[]` (Enabled: `True`). {{(pid=81437) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:07:09.140747 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-faed3037-717d-448f-aeed-a5556a9e377f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:07:09.240221 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b608033-bc84-4aa8-8e23-13bda68a35fa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:07:09.241923 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b608033-bc84-4aa8-8e23-13bda68a35fa admin admin] REQUEST_METHOD: `DELETE` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:07:09.242204 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b608033-bc84-4aa8-8e23-13bda68a35fa admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:07:09.242474 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b608033-bc84-4aa8-8e23-13bda68a35fa admin admin] PATH_INFO: `/v3/projects/b8139ba8cf42465bad4369b51591a32e` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:07:09.246650 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b608033-bc84-4aa8-8e23-13bda68a35fa admin admin] RBAC: Authorizing `identity:delete_project(project_id=b8139ba8cf42465bad4369b51591a32e)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:07:09.253784 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b608033-bc84-4aa8-8e23-13bda68a35fa admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:07:09.350650 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 877/1723] 10.222.0.50 () {58 vars in 958 bytes} [Tue Sep 13 16:07:09 2022] POST /identity/v3/auth/tokens => generated 3248 bytes in 332 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 13 16:07:09.352328 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-faed3037-717d-448f-aeed-a5556a9e377f None None] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:07:09.379881 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-faed3037-717d-448f-aeed-a5556a9e377f None None] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:07:09.382248 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-4b608033-bc84-4aa8-8e23-13bda68a35fa admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'b8139ba8cf42465bad4369b51591a32e', 'request_id': 'req-4b608033-bc84-4aa8-8e23-13bda68a35fa'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:07:09.402152 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-4b608033-bc84-4aa8-8e23-13bda68a35fa admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'b8139ba8cf42465bad4369b51591a32e', 'request_id': 'req-4b608033-bc84-4aa8-8e23-13bda68a35fa'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:07:09.413392 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-4b608033-bc84-4aa8-8e23-13bda68a35fa admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'b8139ba8cf42465bad4369b51591a32e', 'request_id': 'req-4b608033-bc84-4aa8-8e23-13bda68a35fa'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:07:09.419042 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=89,keystone:INSERT=4,keystone:DELETE=7 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:07:09.452613 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-4b608033-bc84-4aa8-8e23-13bda68a35fa admin admin] The token cache is being invalidate because project b8139ba8cf42465bad4369b51591a32e was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81438) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 13 16:07:09.452913 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-4b608033-bc84-4aa8-8e23-13bda68a35fa admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:07:09.457628 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-4b608033-bc84-4aa8-8e23-13bda68a35fa admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:07:09.461888 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 847/1724] 10.222.0.50 () {58 vars in 1233 bytes} [Tue Sep 13 16:07:08 2022] DELETE /identity/v3/projects/b8139ba8cf42465bad4369b51591a32e => generated 0 bytes in 513 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 16:07:09.466764 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-032f33af-42d0-4a75-8cf0-3de9c04c4df8 None None] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:07:09.467089 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-032f33af-42d0-4a75-8cf0-3de9c04c4df8 None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:07:09.467403 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-032f33af-42d0-4a75-8cf0-3de9c04c4df8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:07:09.472899 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:07:09.522641 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.auth.core [None req-032f33af-42d0-4a75-8cf0-3de9c04c4df8 None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81438) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:07:09.524212 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-032f33af-42d0-4a75-8cf0-3de9c04c4df8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:07:09.550937 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52d65030-db49-4689-b02f-21a3c23203f1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:07:09.551331 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52d65030-db49-4689-b02f-21a3c23203f1 admin admin] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:07:09.552971 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52d65030-db49-4689-b02f-21a3c23203f1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:07:09.553591 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52d65030-db49-4689-b02f-21a3c23203f1 admin admin] PATH_INFO: `/v3/projects` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:07:09.555960 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52d65030-db49-4689-b02f-21a3c23203f1 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:07:09.561937 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52d65030-db49-4689-b02f-21a3c23203f1 admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:07:09.632440 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 878/1725] 10.222.0.50 () {60 vars in 1153 bytes} [Tue Sep 13 16:07:09 2022] POST /identity/v3/projects => generated 373 bytes in 280 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 16:07:09.635792 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52d65030-db49-4689-b02f-21a3c23203f1 admin admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:07:09.641426 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-52d65030-db49-4689-b02f-21a3c23203f1 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:07:09.686571 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 848/1726] 10.222.0.50 () {58 vars in 958 bytes} [Tue Sep 13 16:07:09 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 220 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:07:09.695749 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78767df3-9320-410c-99ef-48257c1b1f89 None None] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:07:09.695933 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78767df3-9320-410c-99ef-48257c1b1f89 None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:07:09.696139 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78767df3-9320-410c-99ef-48257c1b1f89 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:07:09.745096 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.auth.core [None req-78767df3-9320-410c-99ef-48257c1b1f89 None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81438) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:07:09.748730 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-78767df3-9320-410c-99ef-48257c1b1f89 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:07:09.815548 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52d65030-db49-4689-b02f-21a3c23203f1 admin admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:07:09.829410 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4095acc-60a2-4c54-97fe-ec9703d953d1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:07:09.832307 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4095acc-60a2-4c54-97fe-ec9703d953d1 admin admin] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:07:09.832933 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4095acc-60a2-4c54-97fe-ec9703d953d1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:07:09.833201 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4095acc-60a2-4c54-97fe-ec9703d953d1 admin admin] PATH_INFO: `/v3/users` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:07:09.835698 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4095acc-60a2-4c54-97fe-ec9703d953d1 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:07:09.842181 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4095acc-60a2-4c54-97fe-ec9703d953d1 admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:07:09.904632 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 879/1727] 10.222.0.50 () {60 vars in 1144 bytes} [Tue Sep 13 16:07:09 2022] POST /identity/v3/users => generated 293 bytes in 267 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 16:07:09.906427 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4095acc-60a2-4c54-97fe-ec9703d953d1 admin admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:07:09.942034 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4095acc-60a2-4c54-97fe-ec9703d953d1 admin admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:07:09.947928 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 849/1728] 10.222.0.50 () {58 vars in 958 bytes} [Tue Sep 13 16:07:09 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 251 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:07:09.950789 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78767df3-9320-410c-99ef-48257c1b1f89 None None] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:07:09.964086 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e580533-af63-4f63-a470-029a27b327cf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:07:09.964748 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e580533-af63-4f63-a470-029a27b327cf admin admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:07:09.965163 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e580533-af63-4f63-a470-029a27b327cf admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:07:09.965352 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e580533-af63-4f63-a470-029a27b327cf admin admin] PATH_INFO: `/v3/roles` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:07:09.966008 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e580533-af63-4f63-a470-029a27b327cf admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:07:09.983908 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78767df3-9320-410c-99ef-48257c1b1f89 None None] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:07:09.988104 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e580533-af63-4f63-a470-029a27b327cf admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:07:09.989395 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c62154a-cb01-4629-9fc3-ce8603136982 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:07:09.990036 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c62154a-cb01-4629-9fc3-ce8603136982 admin admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:07:09.990321 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c62154a-cb01-4629-9fc3-ce8603136982 admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:07:09.990596 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c62154a-cb01-4629-9fc3-ce8603136982 admin admin] PATH_INFO: `/v3/domains` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:07:09.991599 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c62154a-cb01-4629-9fc3-ce8603136982 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81438) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 16:07:09.991599 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c62154a-cb01-4629-9fc3-ce8603136982 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:07:09.993541 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c62154a-cb01-4629-9fc3-ce8603136982 admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:07:10.014545 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 850/1729] 10.222.0.50 () {58 vars in 1153 bytes} [Tue Sep 13 16:07:09 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 60 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 16:07:10.016174 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c62154a-cb01-4629-9fc3-ce8603136982 admin admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:07:10.026647 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 880/1730] 10.222.0.50 () {58 vars in 1122 bytes} [Tue Sep 13 16:07:09 2022] GET /identity/v3/roles => generated 1420 bytes in 118 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 16:07:10.029735 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e580533-af63-4f63-a470-029a27b327cf admin admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:07:10.055236 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c62154a-cb01-4629-9fc3-ce8603136982 admin admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:07:10.091605 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7956084a-c051-4886-84c3-6c1df1c40e2d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:07:10.092287 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7956084a-c051-4886-84c3-6c1df1c40e2d admin admin] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:07:10.092567 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7956084a-c051-4886-84c3-6c1df1c40e2d admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:07:10.092838 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7956084a-c051-4886-84c3-6c1df1c40e2d admin admin] PATH_INFO: `/v3/projects` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:07:10.093480 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7956084a-c051-4886-84c3-6c1df1c40e2d admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:07:10.095431 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7956084a-c051-4886-84c3-6c1df1c40e2d admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:07:10.113755 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e580533-af63-4f63-a470-029a27b327cf admin admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:07:10.125039 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a92c10fb-a0a6-4100-9290-e269a8e56c97 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:07:10.127574 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a92c10fb-a0a6-4100-9290-e269a8e56c97 admin admin] REQUEST_METHOD: `PUT` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:07:10.127856 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a92c10fb-a0a6-4100-9290-e269a8e56c97 admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:07:10.128130 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a92c10fb-a0a6-4100-9290-e269a8e56c97 admin admin] PATH_INFO: `/v3/projects/cef6de435b554306a7e68ac6918f8244/users/680f53a2509242c8b5143a8c6070c275/roles/031d4e1a23e34f71858fc4a6dfbd4ccb` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:07:10.208384 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a92c10fb-a0a6-4100-9290-e269a8e56c97 admin admin] RBAC: Authorizing `identity:create_grant(project_id=cef6de435b554306a7e68ac6918f8244, user_id=680f53a2509242c8b5143a8c6070c275, role_id=031d4e1a23e34f71858fc4a6dfbd4ccb)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:07:10.211333 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a92c10fb-a0a6-4100-9290-e269a8e56c97 admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:07:10.225539 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 851/1731] 10.222.0.50 () {60 vars in 1153 bytes} [Tue Sep 13 16:07:10 2022] POST /identity/v3/projects => generated 371 bytes in 210 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 16:07:10.227004 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f21732e-624e-4fdd-a1e3-813b01cc13c3 None None] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:07:10.228459 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f21732e-624e-4fdd-a1e3-813b01cc13c3 None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:07:10.231075 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f21732e-624e-4fdd-a1e3-813b01cc13c3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:07:10.264196 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 881/1732] 10.222.0.50 () {60 vars in 1483 bytes} [Tue Sep 13 16:07:10 2022] PUT /identity/v3/projects/cef6de435b554306a7e68ac6918f8244/users/680f53a2509242c8b5143a8c6070c275/roles/031d4e1a23e34f71858fc4a6dfbd4ccb => generated 0 bytes in 233 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 16:07:10.265213 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a92c10fb-a0a6-4100-9290-e269a8e56c97 admin admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:07:10.275299 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-a92c10fb-a0a6-4100-9290-e269a8e56c97 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:07:10.315013 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.auth.core [None req-1f21732e-624e-4fdd-a1e3-813b01cc13c3 None None] MFA Rules not processed for user `6d31b2ec391441f7bab3546d2354b0a1`. Rule list: `[]` (Enabled: `True`). {{(pid=81438) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:07:10.316280 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-1f21732e-624e-4fdd-a1e3-813b01cc13c3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:07:10.446250 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:07:10.478766 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a92c10fb-a0a6-4100-9290-e269a8e56c97 admin admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:07:10.488542 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab1c62a4-62b3-46c3-afcc-72bc25146a55 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:07:10.499144 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab1c62a4-62b3-46c3-afcc-72bc25146a55 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:07:10.499871 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab1c62a4-62b3-46c3-afcc-72bc25146a55 admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:07:10.500222 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab1c62a4-62b3-46c3-afcc-72bc25146a55 admin admin] PATH_INFO: `/v3/projects/d20c4bdf69f641a6958eb4cbdecbe9fa` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:07:10.501550 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab1c62a4-62b3-46c3-afcc-72bc25146a55 admin admin] RBAC: Authorizing `identity:delete_project(project_id=d20c4bdf69f641a6958eb4cbdecbe9fa)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:07:10.510148 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab1c62a4-62b3-46c3-afcc-72bc25146a55 admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:07:10.517311 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 852/1733] 10.222.0.50 () {58 vars in 958 bytes} [Tue Sep 13 16:07:10 2022] POST /identity/v3/auth/tokens => generated 3268 bytes in 291 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:07:10.518337 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f21732e-624e-4fdd-a1e3-813b01cc13c3 None None] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:07:10.559286 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f21732e-624e-4fdd-a1e3-813b01cc13c3 None None] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:07:10.583403 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:07:10.586480 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63bdaae8-c515-41b5-a896-0d5bee89e28c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:07:10.587547 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63bdaae8-c515-41b5-a896-0d5bee89e28c admin admin] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:07:10.588132 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63bdaae8-c515-41b5-a896-0d5bee89e28c admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:07:10.588350 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63bdaae8-c515-41b5-a896-0d5bee89e28c admin admin] PATH_INFO: `/v3/users` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:07:10.589022 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63bdaae8-c515-41b5-a896-0d5bee89e28c admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:07:10.597083 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63bdaae8-c515-41b5-a896-0d5bee89e28c admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:07:10.687944 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 853/1734] 10.222.0.50 () {60 vars in 1144 bytes} [Tue Sep 13 16:07:10 2022] POST /identity/v3/users => generated 286 bytes in 167 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 16:07:10.687944 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63bdaae8-c515-41b5-a896-0d5bee89e28c admin admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:07:10.690812 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-63bdaae8-c515-41b5-a896-0d5bee89e28c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:07:10.696486 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-ab1c62a4-62b3-46c3-afcc-72bc25146a55 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'd20c4bdf69f641a6958eb4cbdecbe9fa', 'request_id': 'req-ab1c62a4-62b3-46c3-afcc-72bc25146a55'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:07:10.715508 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-ab1c62a4-62b3-46c3-afcc-72bc25146a55 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'd20c4bdf69f641a6958eb4cbdecbe9fa', 'request_id': 'req-ab1c62a4-62b3-46c3-afcc-72bc25146a55'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:07:10.728122 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-ab1c62a4-62b3-46c3-afcc-72bc25146a55 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'd20c4bdf69f641a6958eb4cbdecbe9fa', 'request_id': 'req-ab1c62a4-62b3-46c3-afcc-72bc25146a55'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:07:10.754482 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-ab1c62a4-62b3-46c3-afcc-72bc25146a55 admin admin] The token cache is being invalidate because project d20c4bdf69f641a6958eb4cbdecbe9fa was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81437) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 13 16:07:10.754856 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-ab1c62a4-62b3-46c3-afcc-72bc25146a55 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:07:10.758512 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-ab1c62a4-62b3-46c3-afcc-72bc25146a55 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:07:10.761253 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 882/1735] 10.222.0.50 () {58 vars in 1233 bytes} [Tue Sep 13 16:07:10 2022] DELETE /identity/v3/projects/d20c4bdf69f641a6958eb4cbdecbe9fa => generated 0 bytes in 496 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 16:07:10.762763 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab1c62a4-62b3-46c3-afcc-72bc25146a55 admin admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:07:10.771949 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-ab1c62a4-62b3-46c3-afcc-72bc25146a55 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:07:10.878565 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63bdaae8-c515-41b5-a896-0d5bee89e28c admin admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:07:10.952481 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab1c62a4-62b3-46c3-afcc-72bc25146a55 admin admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:07:10.963171 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f4e1c23-c27e-4281-b9bc-cdff281e99c7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:07:10.963695 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f4e1c23-c27e-4281-b9bc-cdff281e99c7 admin admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:07:10.963982 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f4e1c23-c27e-4281-b9bc-cdff281e99c7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:07:10.964254 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f4e1c23-c27e-4281-b9bc-cdff281e99c7 admin admin] PATH_INFO: `/v3/roles` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:07:10.964976 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f4e1c23-c27e-4281-b9bc-cdff281e99c7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:07:10.966814 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f4e1c23-c27e-4281-b9bc-cdff281e99c7 admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:07:11.010731 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 883/1736] 10.222.0.50 () {58 vars in 1122 bytes} [Tue Sep 13 16:07:10 2022] GET /identity/v3/roles => generated 1420 bytes in 249 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 16:07:11.015906 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdbd1e02-dd68-4d6e-b67b-a3a2fbc6ad47 None None] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:07:11.018491 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdbd1e02-dd68-4d6e-b67b-a3a2fbc6ad47 None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:07:11.019105 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdbd1e02-dd68-4d6e-b67b-a3a2fbc6ad47 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:07:11.064013 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8957c6ba-4d6d-4198-9b79-2ec92da9cd8d service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '31480b61418340bd88583f034f67412e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:07:11.064713 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8957c6ba-4d6d-4198-9b79-2ec92da9cd8d service placement] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:07:11.064961 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8957c6ba-4d6d-4198-9b79-2ec92da9cd8d service placement] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:07:11.065963 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8957c6ba-4d6d-4198-9b79-2ec92da9cd8d service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:07:11.069491 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-8957c6ba-4d6d-4198-9b79-2ec92da9cd8d service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:07:11.081509 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.auth.core [None req-cdbd1e02-dd68-4d6e-b67b-a3a2fbc6ad47 None None] MFA Rules not processed for user `6d31b2ec391441f7bab3546d2354b0a1`. Rule list: `[]` (Enabled: `True`). {{(pid=81437) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:07:11.086730 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-cdbd1e02-dd68-4d6e-b67b-a3a2fbc6ad47 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:07:11.099481 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8957c6ba-4d6d-4198-9b79-2ec92da9cd8d service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:07:11.101126 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8957c6ba-4d6d-4198-9b79-2ec92da9cd8d service placement] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:07:11.134694 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 854/1737] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 16:07:10 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 449 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 16:07:11.139079 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8957c6ba-4d6d-4198-9b79-2ec92da9cd8d service placement] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:07:11.149776 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-8957c6ba-4d6d-4198-9b79-2ec92da9cd8d service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:07:11.190078 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8957c6ba-4d6d-4198-9b79-2ec92da9cd8d service placement] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:07:11.213169 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d7d9895-675a-4744-9ffd-2c00bd4272b8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:07:11.213483 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d7d9895-675a-4744-9ffd-2c00bd4272b8 admin admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:07:11.213685 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d7d9895-675a-4744-9ffd-2c00bd4272b8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:07:11.213962 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d7d9895-675a-4744-9ffd-2c00bd4272b8 admin admin] PATH_INFO: `/v3/roles` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:07:11.215169 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d7d9895-675a-4744-9ffd-2c00bd4272b8 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:07:11.217102 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d7d9895-675a-4744-9ffd-2c00bd4272b8 admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:07:11.253509 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 884/1738] 10.222.0.50 () {58 vars in 958 bytes} [Tue Sep 13 16:07:11 2022] POST /identity/v3/auth/tokens => generated 3268 bytes in 241 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:07:11.259455 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a7d52d2-ea0d-4712-bb6e-2cd482a21db7 None None] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:07:11.259866 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a7d52d2-ea0d-4712-bb6e-2cd482a21db7 None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:07:11.261247 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a7d52d2-ea0d-4712-bb6e-2cd482a21db7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:07:11.262227 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 855/1739] 10.222.0.50 () {58 vars in 1122 bytes} [Tue Sep 13 16:07:11 2022] GET /identity/v3/roles => generated 1420 bytes in 127 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 16:07:11.263123 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d7d9895-675a-4744-9ffd-2c00bd4272b8 admin admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:07:11.304729 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d7d9895-675a-4744-9ffd-2c00bd4272b8 admin admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:07:11.312910 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.auth.core [None req-1a7d52d2-ea0d-4712-bb6e-2cd482a21db7 None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81437) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:07:11.314383 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-1a7d52d2-ea0d-4712-bb6e-2cd482a21db7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:07:11.320428 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b171b80-c61c-4eae-aaa7-721d95cbc27b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:07:11.321423 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b171b80-c61c-4eae-aaa7-721d95cbc27b admin admin] REQUEST_METHOD: `PUT` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:07:11.321709 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b171b80-c61c-4eae-aaa7-721d95cbc27b admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:07:11.324918 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b171b80-c61c-4eae-aaa7-721d95cbc27b admin admin] PATH_INFO: `/v3/projects/cef6de435b554306a7e68ac6918f8244/users/680f53a2509242c8b5143a8c6070c275/roles/031d4e1a23e34f71858fc4a6dfbd4ccb` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:07:11.332968 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b171b80-c61c-4eae-aaa7-721d95cbc27b admin admin] RBAC: Authorizing `identity:create_grant(project_id=cef6de435b554306a7e68ac6918f8244, user_id=680f53a2509242c8b5143a8c6070c275, role_id=031d4e1a23e34f71858fc4a6dfbd4ccb)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:07:11.341584 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 885/1740] 10.222.0.50 () {58 vars in 958 bytes} [Tue Sep 13 16:07:11 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 82 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:07:11.341584 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a7d52d2-ea0d-4712-bb6e-2cd482a21db7 None None] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:07:11.341971 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b171b80-c61c-4eae-aaa7-721d95cbc27b admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:07:11.342058 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-1a7d52d2-ea0d-4712-bb6e-2cd482a21db7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:07:11.371732 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 856/1741] 10.222.0.50 () {60 vars in 1483 bytes} [Tue Sep 13 16:07:11 2022] PUT /identity/v3/projects/cef6de435b554306a7e68ac6918f8244/users/680f53a2509242c8b5143a8c6070c275/roles/031d4e1a23e34f71858fc4a6dfbd4ccb => generated 0 bytes in 109 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 16:07:11.372747 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b171b80-c61c-4eae-aaa7-721d95cbc27b admin admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:07:11.457706 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:07:11.489597 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:07:11.569414 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a7d52d2-ea0d-4712-bb6e-2cd482a21db7 None None] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:07:11.576027 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b171b80-c61c-4eae-aaa7-721d95cbc27b admin admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:07:11.598617 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a14dbf9-2fb7-4b71-be8b-624fa98214fc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:07:11.601920 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a14dbf9-2fb7-4b71-be8b-624fa98214fc admin admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:07:11.602339 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a14dbf9-2fb7-4b71-be8b-624fa98214fc admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:07:11.602826 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a14dbf9-2fb7-4b71-be8b-624fa98214fc admin admin] PATH_INFO: `/v3/roles` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:07:11.604740 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a14dbf9-2fb7-4b71-be8b-624fa98214fc admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:07:11.609352 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a14dbf9-2fb7-4b71-be8b-624fa98214fc admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:07:11.649409 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 857/1742] 10.222.0.50 () {58 vars in 1122 bytes} [Tue Sep 13 16:07:11 2022] GET /identity/v3/roles => generated 1420 bytes in 277 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 16:07:11.651025 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24ced1f3-7c38-4212-996c-40b16f853b79 None None] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:07:11.651508 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24ced1f3-7c38-4212-996c-40b16f853b79 None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:07:11.653047 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24ced1f3-7c38-4212-996c-40b16f853b79 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:07:11.697107 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.auth.core [None req-24ced1f3-7c38-4212-996c-40b16f853b79 None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81438) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:07:11.698622 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-24ced1f3-7c38-4212-996c-40b16f853b79 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:07:11.715743 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 858/1743] 10.222.0.50 () {58 vars in 958 bytes} [Tue Sep 13 16:07:11 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 63 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:07:11.715743 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72f39f36-5aa0-48c5-a00e-3a6d6286fbc1 None None] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:07:11.715743 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72f39f36-5aa0-48c5-a00e-3a6d6286fbc1 None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:07:11.715743 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72f39f36-5aa0-48c5-a00e-3a6d6286fbc1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:07:11.789664 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.auth.core [None req-72f39f36-5aa0-48c5-a00e-3a6d6286fbc1 None None] MFA Rules not processed for user `680f53a2509242c8b5143a8c6070c275`. Rule list: `[]` (Enabled: `True`). {{(pid=81438) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:07:11.791944 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-72f39f36-5aa0-48c5-a00e-3a6d6286fbc1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:07:11.810680 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aedcf1e9-560b-43e7-b181-d2a20410448e service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4e039b46c5e3445bbb39904b00c5430e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'admin', 'ResellerAdmin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:07:11.811621 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aedcf1e9-560b-43e7-b181-d2a20410448e service nova] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:07:11.812009 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aedcf1e9-560b-43e7-b181-d2a20410448e service nova] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:07:11.812512 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aedcf1e9-560b-43e7-b181-d2a20410448e service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:07:11.820452 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-aedcf1e9-560b-43e7-b181-d2a20410448e service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:07:12.080665 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aedcf1e9-560b-43e7-b181-d2a20410448e service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:07:12.091021 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aedcf1e9-560b-43e7-b181-d2a20410448e service nova] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:07:12.122054 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 859/1744] 10.222.0.50 () {58 vars in 958 bytes} [Tue Sep 13 16:07:11 2022] POST /identity/v3/auth/tokens => generated 3317 bytes in 408 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:07:12.123040 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72f39f36-5aa0-48c5-a00e-3a6d6286fbc1 None None] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:07:12.138733 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 886/1745] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 16:07:11 2022] GET /identity/v3/auth/tokens => generated 3268 bytes in 802 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 16:07:12.139549 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aedcf1e9-560b-43e7-b181-d2a20410448e service nova] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:07:12.173426 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72f39f36-5aa0-48c5-a00e-3a6d6286fbc1 None None] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:07:12.180123 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aedcf1e9-560b-43e7-b181-d2a20410448e service nova] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:07:12.201033 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1702f0d6-d312-4ad4-8cd9-817229c8efb3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:07:12.201735 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1702f0d6-d312-4ad4-8cd9-817229c8efb3 admin admin] REQUEST_METHOD: `PUT` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:07:12.202034 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1702f0d6-d312-4ad4-8cd9-817229c8efb3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:07:12.207754 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8362c92c-11d2-4c22-a170-c4660d88a217 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:07:12.208326 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8362c92c-11d2-4c22-a170-c4660d88a217 admin admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:07:12.208608 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8362c92c-11d2-4c22-a170-c4660d88a217 admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:07:12.208895 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8362c92c-11d2-4c22-a170-c4660d88a217 admin admin] PATH_INFO: `/v3/domains` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:07:12.209591 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8362c92c-11d2-4c22-a170-c4660d88a217 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81437) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 16:07:12.209930 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8362c92c-11d2-4c22-a170-c4660d88a217 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:07:12.212566 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1702f0d6-d312-4ad4-8cd9-817229c8efb3 admin admin] PATH_INFO: `/v3/projects/7c2a2507c1314f3ab4fb1b40babae9e5/users/8eeda926f19c43ac85edf6378bfe8504/roles/15db4aaca3264ae29bde38903ba43471` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:07:12.227144 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8362c92c-11d2-4c22-a170-c4660d88a217 admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:07:12.270509 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1702f0d6-d312-4ad4-8cd9-817229c8efb3 admin admin] RBAC: Authorizing `identity:create_grant(project_id=7c2a2507c1314f3ab4fb1b40babae9e5, user_id=8eeda926f19c43ac85edf6378bfe8504, role_id=15db4aaca3264ae29bde38903ba43471)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:07:12.276237 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1702f0d6-d312-4ad4-8cd9-817229c8efb3 admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:07:12.281407 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 887/1746] 10.222.0.50 () {58 vars in 1153 bytes} [Tue Sep 13 16:07:12 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 16:07:12.284862 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8362c92c-11d2-4c22-a170-c4660d88a217 admin admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:07:12.303533 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-8362c92c-11d2-4c22-a170-c4660d88a217 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:07:12.326958 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 860/1747] 10.222.0.50 () {60 vars in 1483 bytes} [Tue Sep 13 16:07:12 2022] PUT /identity/v3/projects/7c2a2507c1314f3ab4fb1b40babae9e5/users/8eeda926f19c43ac85edf6378bfe8504/roles/15db4aaca3264ae29bde38903ba43471 => generated 0 bytes in 204 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 16:07:12.328005 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1702f0d6-d312-4ad4-8cd9-817229c8efb3 admin admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:07:12.491516 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8362c92c-11d2-4c22-a170-c4660d88a217 admin admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:07:12.520347 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:07:12.556554 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1702f0d6-d312-4ad4-8cd9-817229c8efb3 admin admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:07:12.595277 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e00a828-86bd-47d6-a586-3e7a1b765d5d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:07:12.595912 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e00a828-86bd-47d6-a586-3e7a1b765d5d admin admin] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:07:12.596191 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e00a828-86bd-47d6-a586-3e7a1b765d5d admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:07:12.596470 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e00a828-86bd-47d6-a586-3e7a1b765d5d admin admin] PATH_INFO: `/v3/projects` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:07:12.601510 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e00a828-86bd-47d6-a586-3e7a1b765d5d admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:07:12.612252 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e00a828-86bd-47d6-a586-3e7a1b765d5d admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:07:12.673020 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a114ae12-2a01-49c2-b5cf-20ed2e4fefd5 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7fd6b5afd67b48f9b0f0597ccd4cf4df', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:07:12.677994 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a114ae12-2a01-49c2-b5cf-20ed2e4fefd5 service cinder] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:07:12.678283 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a114ae12-2a01-49c2-b5cf-20ed2e4fefd5 service cinder] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:07:12.678563 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a114ae12-2a01-49c2-b5cf-20ed2e4fefd5 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:07:12.692029 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 861/1748] 10.222.0.50 () {60 vars in 1153 bytes} [Tue Sep 13 16:07:12 2022] POST /identity/v3/projects => generated 373 bytes in 364 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 16:07:12.693686 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10bd81e3-b701-4c50-b19c-dabeb251e813 None None] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:07:12.694149 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10bd81e3-b701-4c50-b19c-dabeb251e813 None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:07:12.694633 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10bd81e3-b701-4c50-b19c-dabeb251e813 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:07:12.776517 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.auth.core [None req-10bd81e3-b701-4c50-b19c-dabeb251e813 None None] MFA Rules not processed for user `8eeda926f19c43ac85edf6378bfe8504`. Rule list: `[]` (Enabled: `True`). {{(pid=81438) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:07:12.782885 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-10bd81e3-b701-4c50-b19c-dabeb251e813 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:07:12.798159 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:07:12.939734 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a114ae12-2a01-49c2-b5cf-20ed2e4fefd5 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:07:12.943427 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a114ae12-2a01-49c2-b5cf-20ed2e4fefd5 service cinder] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:07:12.970071 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 888/1749] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 16:07:12 2022] GET /identity/v3/auth/tokens => generated 3317 bytes in 686 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 16:07:12.972990 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e1ca83d-6064-4b55-93c3-f4372603972e None None] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:07:12.973279 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e1ca83d-6064-4b55-93c3-f4372603972e None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:07:12.973558 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e1ca83d-6064-4b55-93c3-f4372603972e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:07:13.031143 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.auth.core [None req-5e1ca83d-6064-4b55-93c3-f4372603972e None None] MFA Rules not processed for user `e1cbafa1d3c244c5906e475494b39d4c`. Rule list: `[]` (Enabled: `True`). {{(pid=81437) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:07:13.038924 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-5e1ca83d-6064-4b55-93c3-f4372603972e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:07:13.069014 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 862/1750] 10.222.0.50 () {58 vars in 958 bytes} [Tue Sep 13 16:07:12 2022] POST /identity/v3/auth/tokens => generated 3248 bytes in 376 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:07:13.073031 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10bd81e3-b701-4c50-b19c-dabeb251e813 None None] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:07:13.276570 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 889/1751] 10.222.0.50 () {58 vars in 958 bytes} [Tue Sep 13 16:07:12 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 306 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:07:13.278047 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e1ca83d-6064-4b55-93c3-f4372603972e None None] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:07:13.324664 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10bd81e3-b701-4c50-b19c-dabeb251e813 None None] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:07:13.334880 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43f01262-a2c3-449a-8d00-93e2ccf4fa6e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:07:13.335550 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43f01262-a2c3-449a-8d00-93e2ccf4fa6e admin admin] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:07:13.335830 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43f01262-a2c3-449a-8d00-93e2ccf4fa6e admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:07:13.336109 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43f01262-a2c3-449a-8d00-93e2ccf4fa6e admin admin] PATH_INFO: `/v3/users` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:07:13.336751 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43f01262-a2c3-449a-8d00-93e2ccf4fa6e admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:07:13.338751 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43f01262-a2c3-449a-8d00-93e2ccf4fa6e admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:07:13.408869 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 863/1752] 10.222.0.50 () {60 vars in 1144 bytes} [Tue Sep 13 16:07:13 2022] POST /identity/v3/users => generated 287 bytes in 339 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 16:07:13.414144 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43f01262-a2c3-449a-8d00-93e2ccf4fa6e admin admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:07:13.554656 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e1ca83d-6064-4b55-93c3-f4372603972e None None] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:07:13.572046 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-582b34ca-ed46-4d76-8f04-aecbbd09bc0b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:07:13.572688 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-582b34ca-ed46-4d76-8f04-aecbbd09bc0b admin admin] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:07:13.573355 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-582b34ca-ed46-4d76-8f04-aecbbd09bc0b admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:07:13.573355 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-582b34ca-ed46-4d76-8f04-aecbbd09bc0b admin admin] PATH_INFO: `/v3/projects` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:07:13.574579 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-582b34ca-ed46-4d76-8f04-aecbbd09bc0b admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:07:13.579744 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-582b34ca-ed46-4d76-8f04-aecbbd09bc0b admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:07:13.612765 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43f01262-a2c3-449a-8d00-93e2ccf4fa6e admin admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:07:13.623757 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97c6a25d-ca2a-4fe3-9001-6c8b0cc6f104 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7fd6b5afd67b48f9b0f0597ccd4cf4df', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:07:13.625194 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97c6a25d-ca2a-4fe3-9001-6c8b0cc6f104 service cinder] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:07:13.625194 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97c6a25d-ca2a-4fe3-9001-6c8b0cc6f104 service cinder] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:07:13.625194 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97c6a25d-ca2a-4fe3-9001-6c8b0cc6f104 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:07:13.627688 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:07:13.630286 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-97c6a25d-ca2a-4fe3-9001-6c8b0cc6f104 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:07:13.656200 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 890/1753] 10.222.0.50 () {60 vars in 1153 bytes} [Tue Sep 13 16:07:13 2022] POST /identity/v3/projects => generated 373 bytes in 379 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 16:07:13.656805 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-582b34ca-ed46-4d76-8f04-aecbbd09bc0b admin admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:07:13.769654 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:07:13.803532 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97c6a25d-ca2a-4fe3-9001-6c8b0cc6f104 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:07:13.808704 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97c6a25d-ca2a-4fe3-9001-6c8b0cc6f104 service cinder] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:07:13.839360 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 864/1754] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 16:07:13 2022] GET /identity/v3/auth/tokens => generated 3248 bytes in 429 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 16:07:13.840951 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97c6a25d-ca2a-4fe3-9001-6c8b0cc6f104 service cinder] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:07:13.860658 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-582b34ca-ed46-4d76-8f04-aecbbd09bc0b admin admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:07:13.865609 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97c6a25d-ca2a-4fe3-9001-6c8b0cc6f104 service cinder] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:07:13.887474 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fc40a55-58e0-46b0-921b-082616bb3548 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:07:13.888102 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fc40a55-58e0-46b0-921b-082616bb3548 admin admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:07:13.888381 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fc40a55-58e0-46b0-921b-082616bb3548 admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:07:13.888652 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fc40a55-58e0-46b0-921b-082616bb3548 admin admin] PATH_INFO: `/v3/roles` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:07:13.891156 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f48e3f5-ad33-4028-bf70-f2cbf241afd8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:07:13.891778 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f48e3f5-ad33-4028-bf70-f2cbf241afd8 admin admin] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:07:13.892055 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f48e3f5-ad33-4028-bf70-f2cbf241afd8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:07:13.892325 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f48e3f5-ad33-4028-bf70-f2cbf241afd8 admin admin] PATH_INFO: `/v3/users` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:07:13.893306 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fc40a55-58e0-46b0-921b-082616bb3548 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:07:13.895508 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f48e3f5-ad33-4028-bf70-f2cbf241afd8 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:07:13.896545 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fc40a55-58e0-46b0-921b-082616bb3548 admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:07:13.898543 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f48e3f5-ad33-4028-bf70-f2cbf241afd8 admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:07:13.925234 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 891/1755] 10.222.0.50 () {58 vars in 1122 bytes} [Tue Sep 13 16:07:13 2022] GET /identity/v3/roles => generated 1420 bytes in 269 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 16:07:13.926540 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3999000-c7cd-49da-8ccd-629eaaf8c6f0 None None] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:07:13.926827 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3999000-c7cd-49da-8ccd-629eaaf8c6f0 None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:07:13.927128 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3999000-c7cd-49da-8ccd-629eaaf8c6f0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:07:13.951339 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 865/1756] 10.222.0.50 () {60 vars in 1144 bytes} [Tue Sep 13 16:07:13 2022] POST /identity/v3/users => generated 293 bytes in 109 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 16:07:13.952160 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f48e3f5-ad33-4028-bf70-f2cbf241afd8 admin admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:07:13.977635 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f48e3f5-ad33-4028-bf70-f2cbf241afd8 admin admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:07:13.979556 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.auth.core [None req-a3999000-c7cd-49da-8ccd-629eaaf8c6f0 None None] MFA Rules not processed for user `c0fd4bbe430b4a8888f3be4b23982c5a`. Rule list: `[]` (Enabled: `True`). {{(pid=81437) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:07:13.984145 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-a3999000-c7cd-49da-8ccd-629eaaf8c6f0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:07:13.998156 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-447471a3-93f0-4c13-9a6a-a502ef4bae80 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:07:13.999130 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-447471a3-93f0-4c13-9a6a-a502ef4bae80 admin admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:07:14.001175 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-447471a3-93f0-4c13-9a6a-a502ef4bae80 admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:07:14.002097 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-447471a3-93f0-4c13-9a6a-a502ef4bae80 admin admin] PATH_INFO: `/v3/roles` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:07:14.003590 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-447471a3-93f0-4c13-9a6a-a502ef4bae80 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:07:14.009655 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-447471a3-93f0-4c13-9a6a-a502ef4bae80 admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:07:14.038871 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 866/1757] 10.222.0.50 () {58 vars in 1122 bytes} [Tue Sep 13 16:07:13 2022] GET /identity/v3/roles => generated 1420 bytes in 89 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 16:07:14.040085 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-447471a3-93f0-4c13-9a6a-a502ef4bae80 admin admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:07:14.079186 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-447471a3-93f0-4c13-9a6a-a502ef4bae80 admin admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:07:14.091942 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-daa7e4fe-3d58-4263-8463-b25fb6df4fe6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:07:14.095268 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daa7e4fe-3d58-4263-8463-b25fb6df4fe6 admin admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:07:14.095788 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daa7e4fe-3d58-4263-8463-b25fb6df4fe6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:07:14.096577 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daa7e4fe-3d58-4263-8463-b25fb6df4fe6 admin admin] PATH_INFO: `/v3/roles` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:07:14.099157 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-daa7e4fe-3d58-4263-8463-b25fb6df4fe6 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:07:14.100973 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-daa7e4fe-3d58-4263-8463-b25fb6df4fe6 admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:07:14.161315 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 867/1758] 10.222.0.50 () {58 vars in 1122 bytes} [Tue Sep 13 16:07:14 2022] GET /identity/v3/roles => generated 1420 bytes in 121 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 16:07:14.162023 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-daa7e4fe-3d58-4263-8463-b25fb6df4fe6 admin admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:07:14.176706 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 892/1759] 10.222.0.50 () {58 vars in 958 bytes} [Tue Sep 13 16:07:13 2022] POST /identity/v3/auth/tokens => generated 3266 bytes in 250 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:07:14.177768 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c2c9543-2fc1-4e94-b067-8b2e4c626478 None None] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:07:14.178074 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c2c9543-2fc1-4e94-b067-8b2e4c626478 None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:07:14.178316 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c2c9543-2fc1-4e94-b067-8b2e4c626478 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:07:14.188969 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-daa7e4fe-3d58-4263-8463-b25fb6df4fe6 admin admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:07:14.203450 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-469f8e02-6edc-4537-bad4-9db1f664868e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:07:14.204148 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-469f8e02-6edc-4537-bad4-9db1f664868e admin admin] REQUEST_METHOD: `PUT` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:07:14.204425 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-469f8e02-6edc-4537-bad4-9db1f664868e admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:07:14.204701 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-469f8e02-6edc-4537-bad4-9db1f664868e admin admin] PATH_INFO: `/v3/projects/0f1ad90d23904e05b750d8ba0147119e/users/c1eee7151f44462eac88d35859366e2f/roles/15db4aaca3264ae29bde38903ba43471` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:07:14.263195 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.auth.core [None req-5c2c9543-2fc1-4e94-b067-8b2e4c626478 None None] MFA Rules not processed for user `6d31b2ec391441f7bab3546d2354b0a1`. Rule list: `[]` (Enabled: `True`). {{(pid=81437) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:07:14.271936 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-5c2c9543-2fc1-4e94-b067-8b2e4c626478 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:07:14.283493 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-469f8e02-6edc-4537-bad4-9db1f664868e admin admin] RBAC: Authorizing `identity:create_grant(project_id=0f1ad90d23904e05b750d8ba0147119e, user_id=c1eee7151f44462eac88d35859366e2f, role_id=15db4aaca3264ae29bde38903ba43471)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:07:14.290233 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-469f8e02-6edc-4537-bad4-9db1f664868e admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:07:14.323103 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 868/1760] 10.222.0.50 () {60 vars in 1483 bytes} [Tue Sep 13 16:07:14 2022] PUT /identity/v3/projects/0f1ad90d23904e05b750d8ba0147119e/users/c1eee7151f44462eac88d35859366e2f/roles/15db4aaca3264ae29bde38903ba43471 => generated 0 bytes in 161 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 16:07:14.326273 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-469f8e02-6edc-4537-bad4-9db1f664868e admin admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:07:14.421491 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 893/1761] 10.222.0.50 () {58 vars in 958 bytes} [Tue Sep 13 16:07:14 2022] POST /identity/v3/auth/tokens => generated 3268 bytes in 244 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:07:14.423834 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-329eac34-e1b3-41d1-8786-0b57a4d40037 None None] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:07:14.424126 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-329eac34-e1b3-41d1-8786-0b57a4d40037 None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:07:14.425809 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-329eac34-e1b3-41d1-8786-0b57a4d40037 None None] PATH_INFO: `/` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:07:14.430008 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 894/1762] 10.222.0.50 () {54 vars in 912 bytes} [Tue Sep 13 16:07:14 2022] GET /identity => generated 270 bytes in 8 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 16:07:14.434240 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9a06fb2-49fb-4288-aec8-d77b6cee60ff None None] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:07:14.434528 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9a06fb2-49fb-4288-aec8-d77b6cee60ff None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:07:14.434798 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9a06fb2-49fb-4288-aec8-d77b6cee60ff None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:07:14.478102 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-469f8e02-6edc-4537-bad4-9db1f664868e admin admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:07:14.490349 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b841a6d4-d6d4-4ff9-8b82-bf16b726498e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:07:14.491058 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b841a6d4-d6d4-4ff9-8b82-bf16b726498e admin admin] REQUEST_METHOD: `PUT` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:07:14.491338 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b841a6d4-d6d4-4ff9-8b82-bf16b726498e admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:07:14.491609 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b841a6d4-d6d4-4ff9-8b82-bf16b726498e admin admin] PATH_INFO: `/v3/projects/e46372a21ef248e38fcdf225cd29af15/users/e2b69b7af38d4904ba1ce4a192d423f7/roles/031d4e1a23e34f71858fc4a6dfbd4ccb` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:07:14.505328 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.auth.core [None req-a9a06fb2-49fb-4288-aec8-d77b6cee60ff None None] MFA Rules not processed for user `c1eee7151f44462eac88d35859366e2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81437) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:07:14.506460 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-a9a06fb2-49fb-4288-aec8-d77b6cee60ff None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:07:14.524591 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b841a6d4-d6d4-4ff9-8b82-bf16b726498e admin admin] RBAC: Authorizing `identity:create_grant(project_id=e46372a21ef248e38fcdf225cd29af15, user_id=e2b69b7af38d4904ba1ce4a192d423f7, role_id=031d4e1a23e34f71858fc4a6dfbd4ccb)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:07:14.530010 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b841a6d4-d6d4-4ff9-8b82-bf16b726498e admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:07:14.596640 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 869/1763] 10.222.0.50 () {60 vars in 1483 bytes} [Tue Sep 13 16:07:14 2022] PUT /identity/v3/projects/e46372a21ef248e38fcdf225cd29af15/users/e2b69b7af38d4904ba1ce4a192d423f7/roles/031d4e1a23e34f71858fc4a6dfbd4ccb => generated 0 bytes in 270 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 16:07:14.599893 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f44fd36-aeae-4e1c-9211-72f8ee2ed608 None None] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:07:14.600127 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f44fd36-aeae-4e1c-9211-72f8ee2ed608 None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:07:14.602067 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f44fd36-aeae-4e1c-9211-72f8ee2ed608 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:07:14.633578 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:07:14.672890 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.auth.core [None req-3f44fd36-aeae-4e1c-9211-72f8ee2ed608 None None] MFA Rules not processed for user `2f436f07939d4855a21b195bc7566244`. Rule list: `[]` (Enabled: `True`). {{(pid=81438) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:07:14.674414 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-3f44fd36-aeae-4e1c-9211-72f8ee2ed608 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:07:14.739446 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 895/1764] 10.222.0.50 () {58 vars in 958 bytes} [Tue Sep 13 16:07:14 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 308 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:07:14.742116 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9a06fb2-49fb-4288-aec8-d77b6cee60ff None None] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:07:14.790081 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 870/1765] 10.222.0.50 () {58 vars in 1009 bytes} [Tue Sep 13 16:07:14 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 193 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:07:14.790770 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f44fd36-aeae-4e1c-9211-72f8ee2ed608 None None] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:07:14.793455 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-3f44fd36-aeae-4e1c-9211-72f8ee2ed608 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:07:14.819435 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:07:14.839505 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9a06fb2-49fb-4288-aec8-d77b6cee60ff None None] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:07:14.892183 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f44fd36-aeae-4e1c-9211-72f8ee2ed608 None None] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:07:14.898491 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b83df016-439e-46e8-9dc6-7493c4ddd042 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7fd6b5afd67b48f9b0f0597ccd4cf4df', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:07:14.902778 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b83df016-439e-46e8-9dc6-7493c4ddd042 service cinder] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:07:14.903873 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b83df016-439e-46e8-9dc6-7493c4ddd042 service cinder] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:07:14.904155 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b83df016-439e-46e8-9dc6-7493c4ddd042 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:07:14.964649 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89b9079a-7055-4b55-8c65-dfe51ec4c6cb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:07:14.965577 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89b9079a-7055-4b55-8c65-dfe51ec4c6cb admin admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:07:14.966524 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89b9079a-7055-4b55-8c65-dfe51ec4c6cb admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:07:14.968541 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89b9079a-7055-4b55-8c65-dfe51ec4c6cb admin admin] PATH_INFO: `/v3/roles` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:07:14.969327 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89b9079a-7055-4b55-8c65-dfe51ec4c6cb admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:07:14.972867 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89b9079a-7055-4b55-8c65-dfe51ec4c6cb admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:07:15.007787 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 896/1766] 10.222.0.50 () {58 vars in 1122 bytes} [Tue Sep 13 16:07:14 2022] GET /identity/v3/roles => generated 1420 bytes in 268 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 16:07:15.009626 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89b9079a-7055-4b55-8c65-dfe51ec4c6cb admin admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:07:15.013440 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-89b9079a-7055-4b55-8c65-dfe51ec4c6cb admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:07:15.024760 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b83df016-439e-46e8-9dc6-7493c4ddd042 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:07:15.033435 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89b9079a-7055-4b55-8c65-dfe51ec4c6cb admin admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:07:15.033703 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b83df016-439e-46e8-9dc6-7493c4ddd042 service cinder] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:07:15.044638 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff11ee4e-981a-476a-8b41-6b9cb9717c49 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f436f07939d4855a21b195bc7566244', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:07:15.045369 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff11ee4e-981a-476a-8b41-6b9cb9717c49 service neutron] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:07:15.051732 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff11ee4e-981a-476a-8b41-6b9cb9717c49 service neutron] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:07:15.052734 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff11ee4e-981a-476a-8b41-6b9cb9717c49 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:07:15.069530 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff11ee4e-981a-476a-8b41-6b9cb9717c49 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:07:15.072432 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff11ee4e-981a-476a-8b41-6b9cb9717c49 service neutron] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:07:15.080941 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 871/1767] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 16:07:14 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 290 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 16:07:15.082258 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b83df016-439e-46e8-9dc6-7493c4ddd042 service cinder] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:07:15.100406 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 897/1768] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 16:07:15 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 92 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 16:07:15.101245 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff11ee4e-981a-476a-8b41-6b9cb9717c49 service neutron] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:07:15.110865 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b83df016-439e-46e8-9dc6-7493c4ddd042 service cinder] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:07:15.120010 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0ec544f-b540-461d-ad47-e478cda0dadc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:07:15.121171 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0ec544f-b540-461d-ad47-e478cda0dadc admin admin] REQUEST_METHOD: `PUT` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:07:15.121510 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0ec544f-b540-461d-ad47-e478cda0dadc admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:07:15.121796 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0ec544f-b540-461d-ad47-e478cda0dadc admin admin] PATH_INFO: `/v3/projects/e46372a21ef248e38fcdf225cd29af15/users/e2b69b7af38d4904ba1ce4a192d423f7/roles/031d4e1a23e34f71858fc4a6dfbd4ccb` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:07:15.130916 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0ec544f-b540-461d-ad47-e478cda0dadc admin admin] RBAC: Authorizing `identity:create_grant(project_id=e46372a21ef248e38fcdf225cd29af15, user_id=e2b69b7af38d4904ba1ce4a192d423f7, role_id=031d4e1a23e34f71858fc4a6dfbd4ccb)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:07:15.138394 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0ec544f-b540-461d-ad47-e478cda0dadc admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:07:15.139814 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff11ee4e-981a-476a-8b41-6b9cb9717c49 service neutron] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:07:15.152558 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da7904f1-05ec-4d08-b7c8-a8ca25b8c8ed admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:07:15.153394 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da7904f1-05ec-4d08-b7c8-a8ca25b8c8ed admin admin] REQUEST_METHOD: `DELETE` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:07:15.153875 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da7904f1-05ec-4d08-b7c8-a8ca25b8c8ed admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:07:15.154258 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da7904f1-05ec-4d08-b7c8-a8ca25b8c8ed admin admin] PATH_INFO: `/v3/users/c1eee7151f44462eac88d35859366e2f` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:07:15.155826 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da7904f1-05ec-4d08-b7c8-a8ca25b8c8ed admin admin] RBAC: Authorizing `identity:delete_user(user_id=c1eee7151f44462eac88d35859366e2f)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:07:15.158092 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da7904f1-05ec-4d08-b7c8-a8ca25b8c8ed admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:07:15.180031 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 872/1769] 10.222.0.50 () {60 vars in 1483 bytes} [Tue Sep 13 16:07:15 2022] PUT /identity/v3/projects/e46372a21ef248e38fcdf225cd29af15/users/e2b69b7af38d4904ba1ce4a192d423f7/roles/031d4e1a23e34f71858fc4a6dfbd4ccb => generated 0 bytes in 90 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 16:07:15.185551 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a804b8f9-9743-410f-84e4-4192da2d96b6 None None] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:07:15.185841 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a804b8f9-9743-410f-84e4-4192da2d96b6 None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:07:15.186185 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a804b8f9-9743-410f-84e4-4192da2d96b6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:07:15.231992 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.auth.core [None req-a804b8f9-9743-410f-84e4-4192da2d96b6 None None] MFA Rules not processed for user `e2b69b7af38d4904ba1ce4a192d423f7`. Rule list: `[]` (Enabled: `True`). {{(pid=81438) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:07:15.233934 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-a804b8f9-9743-410f-84e4-4192da2d96b6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:07:15.234155 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-da7904f1-05ec-4d08-b7c8-a8ca25b8c8ed admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'c1eee7151f44462eac88d35859366e2f'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:07:15.246980 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-da7904f1-05ec-4d08-b7c8-a8ca25b8c8ed admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'c1eee7151f44462eac88d35859366e2f'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:07:15.276709 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-da7904f1-05ec-4d08-b7c8-a8ca25b8c8ed admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'c1eee7151f44462eac88d35859366e2f'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:07:15.278211 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-da7904f1-05ec-4d08-b7c8-a8ca25b8c8ed admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'c1eee7151f44462eac88d35859366e2f'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:07:15.288002 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-da7904f1-05ec-4d08-b7c8-a8ca25b8c8ed admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'c1eee7151f44462eac88d35859366e2f'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:07:15.301880 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 898/1770] 10.222.0.50 () {58 vars in 1224 bytes} [Tue Sep 13 16:07:15 2022] DELETE /identity/v3/users/c1eee7151f44462eac88d35859366e2f => generated 0 bytes in 201 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 16:07:15.303312 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe5b78f3-6057-4d6c-9584-2f517c72db2a None None] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:07:15.303869 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe5b78f3-6057-4d6c-9584-2f517c72db2a None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:07:15.304253 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe5b78f3-6057-4d6c-9584-2f517c72db2a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:07:15.405684 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:07:15.425171 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.auth.core [None req-fe5b78f3-6057-4d6c-9584-2f517c72db2a None None] MFA Rules not processed for user `f8364b1a190d4218bf353a89bdb81448`. Rule list: `[]` (Enabled: `True`). {{(pid=81437) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:07:15.428972 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-fe5b78f3-6057-4d6c-9584-2f517c72db2a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:07:15.449582 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 873/1771] 10.222.0.50 () {58 vars in 958 bytes} [Tue Sep 13 16:07:15 2022] POST /identity/v3/auth/tokens => generated 3317 bytes in 266 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:07:15.467163 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-733c87f7-74da-4d9f-8864-aa327cbc9604 None None] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:07:15.467565 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-733c87f7-74da-4d9f-8864-aa327cbc9604 None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:07:15.467850 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-733c87f7-74da-4d9f-8864-aa327cbc9604 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:07:15.523180 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.auth.core [None req-733c87f7-74da-4d9f-8864-aa327cbc9604 None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81438) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:07:15.528176 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-733c87f7-74da-4d9f-8864-aa327cbc9604 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:07:15.634610 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 899/1772] 10.222.0.50 () {58 vars in 1009 bytes} [Tue Sep 13 16:07:15 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 332 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:07:15.636360 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe5b78f3-6057-4d6c-9584-2f517c72db2a None None] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:07:15.641801 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-fe5b78f3-6057-4d6c-9584-2f517c72db2a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:07:15.692014 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:07:15.758040 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 874/1773] 10.222.0.50 () {58 vars in 958 bytes} [Tue Sep 13 16:07:15 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 306 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:07:15.789584 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe5b78f3-6057-4d6c-9584-2f517c72db2a None None] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:07:15.802254 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a34a29e-c200-4f38-a565-22c01d09148d service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7fd6b5afd67b48f9b0f0597ccd4cf4df', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:07:15.803088 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a34a29e-c200-4f38-a565-22c01d09148d service cinder] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:07:15.803380 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a34a29e-c200-4f38-a565-22c01d09148d service cinder] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:07:15.803864 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-733c87f7-74da-4d9f-8864-aa327cbc9604 None None] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:07:15.805536 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-733c87f7-74da-4d9f-8864-aa327cbc9604 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:07:15.805794 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a34a29e-c200-4f38-a565-22c01d09148d service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:07:15.987466 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-733c87f7-74da-4d9f-8864-aa327cbc9604 None None] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:07:15.994114 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40cf3aa2-6964-4651-a89f-010994f2f6dc service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f436f07939d4855a21b195bc7566244', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:07:15.994549 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40cf3aa2-6964-4651-a89f-010994f2f6dc service neutron] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:07:15.994829 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40cf3aa2-6964-4651-a89f-010994f2f6dc service neutron] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:07:15.995148 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40cf3aa2-6964-4651-a89f-010994f2f6dc service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:07:16.037594 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40cf3aa2-6964-4651-a89f-010994f2f6dc service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:07:16.045331 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40cf3aa2-6964-4651-a89f-010994f2f6dc service neutron] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:07:16.098474 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 875/1774] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 16:07:15 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 296 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 16:07:16.099501 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40cf3aa2-6964-4651-a89f-010994f2f6dc service neutron] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:07:16.101779 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-40cf3aa2-6964-4651-a89f-010994f2f6dc service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:07:16.128939 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a34a29e-c200-4f38-a565-22c01d09148d service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:07:16.134426 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a34a29e-c200-4f38-a565-22c01d09148d service cinder] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:07:16.183204 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 900/1775] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 16:07:15 2022] GET /identity/v3/auth/tokens => generated 3317 bytes in 548 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 16:07:16.409740 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40cf3aa2-6964-4651-a89f-010994f2f6dc service neutron] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:07:16.424992 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec0ebe26-2cce-4134-83f7-4d8070588006 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b00ab0d6321d483188bc69deaddcdbbc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:07:16.428671 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec0ebe26-2cce-4134-83f7-4d8070588006 service swift] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:07:16.429116 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec0ebe26-2cce-4134-83f7-4d8070588006 service swift] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:07:16.430540 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec0ebe26-2cce-4134-83f7-4d8070588006 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:07:16.448835 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec0ebe26-2cce-4134-83f7-4d8070588006 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:07:16.465083 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec0ebe26-2cce-4134-83f7-4d8070588006 service swift] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:07:16.499458 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 876/1776] 10.222.0.50 () {60 vars in 1384 bytes} [Tue Sep 13 16:07:16 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 400 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Sep 13 16:07:17.286773 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a34a29e-c200-4f38-a565-22c01d09148d service cinder] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:07:17.295350 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-7a34a29e-c200-4f38-a565-22c01d09148d service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:07:17.358406 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a34a29e-c200-4f38-a565-22c01d09148d service cinder] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:07:17.386970 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82022d31-a5dc-4f1f-aec1-2526b74c1b65 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:07:17.396568 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82022d31-a5dc-4f1f-aec1-2526b74c1b65 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:07:17.397684 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82022d31-a5dc-4f1f-aec1-2526b74c1b65 admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:07:17.397992 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82022d31-a5dc-4f1f-aec1-2526b74c1b65 admin admin] PATH_INFO: `/v3/users/f6bfdc6bb6bf442cb43bb1d9b9093ed5` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:07:17.408915 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82022d31-a5dc-4f1f-aec1-2526b74c1b65 admin admin] RBAC: Authorizing `identity:delete_user(user_id=f6bfdc6bb6bf442cb43bb1d9b9093ed5)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:07:17.431617 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82022d31-a5dc-4f1f-aec1-2526b74c1b65 admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:07:17.622762 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=94,keystone:DELETE=6 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:07:17.623610 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec0ebe26-2cce-4134-83f7-4d8070588006 service swift] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:07:17.625747 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-82022d31-a5dc-4f1f-aec1-2526b74c1b65 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'f6bfdc6bb6bf442cb43bb1d9b9093ed5'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:07:17.626737 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-ec0ebe26-2cce-4134-83f7-4d8070588006 service swift] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:07:17.640636 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-82022d31-a5dc-4f1f-aec1-2526b74c1b65 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'f6bfdc6bb6bf442cb43bb1d9b9093ed5'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:07:17.663502 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-82022d31-a5dc-4f1f-aec1-2526b74c1b65 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'f6bfdc6bb6bf442cb43bb1d9b9093ed5'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:07:17.665169 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-82022d31-a5dc-4f1f-aec1-2526b74c1b65 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'f6bfdc6bb6bf442cb43bb1d9b9093ed5'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:07:17.679042 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec0ebe26-2cce-4134-83f7-4d8070588006 service swift] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:07:17.699857 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f08f34d-1f27-43b8-bc1f-407df740710e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:07:17.701126 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f08f34d-1f27-43b8-bc1f-407df740710e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:07:17.703205 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f08f34d-1f27-43b8-bc1f-407df740710e admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:07:17.704279 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f08f34d-1f27-43b8-bc1f-407df740710e admin admin] PATH_INFO: `/v3/users/8eeda926f19c43ac85edf6378bfe8504` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:07:17.707729 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-82022d31-a5dc-4f1f-aec1-2526b74c1b65 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'f6bfdc6bb6bf442cb43bb1d9b9093ed5'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:07:17.712702 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f08f34d-1f27-43b8-bc1f-407df740710e admin admin] RBAC: Authorizing `identity:delete_user(user_id=8eeda926f19c43ac85edf6378bfe8504)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:07:17.718111 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 901/1777] 10.222.0.50 () {58 vars in 1224 bytes} [Tue Sep 13 16:07:17 2022] DELETE /identity/v3/users/f6bfdc6bb6bf442cb43bb1d9b9093ed5 => generated 0 bytes in 432 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 16:07:17.723693 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-366aa649-ae05-4b77-bcdd-f90e074b70c8 None None] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:07:17.724191 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-366aa649-ae05-4b77-bcdd-f90e074b70c8 None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:07:17.724633 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-366aa649-ae05-4b77-bcdd-f90e074b70c8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:07:17.727689 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f08f34d-1f27-43b8-bc1f-407df740710e admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:07:17.747878 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.auth.core [None req-366aa649-ae05-4b77-bcdd-f90e074b70c8 None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81437) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:07:17.749405 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-366aa649-ae05-4b77-bcdd-f90e074b70c8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:07:17.942378 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-6f08f34d-1f27-43b8-bc1f-407df740710e admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '8eeda926f19c43ac85edf6378bfe8504'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:07:17.966388 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=91,keystone:DELETE=8,keystone:INSERT=1 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:07:17.981706 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-6f08f34d-1f27-43b8-bc1f-407df740710e admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '8eeda926f19c43ac85edf6378bfe8504'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:07:18.024306 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-6f08f34d-1f27-43b8-bc1f-407df740710e admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '8eeda926f19c43ac85edf6378bfe8504'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:07:18.027578 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-6f08f34d-1f27-43b8-bc1f-407df740710e admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '8eeda926f19c43ac85edf6378bfe8504'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:07:18.045177 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-6f08f34d-1f27-43b8-bc1f-407df740710e admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '8eeda926f19c43ac85edf6378bfe8504'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:07:18.053929 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 877/1778] 10.222.0.50 () {58 vars in 1224 bytes} [Tue Sep 13 16:07:17 2022] DELETE /identity/v3/users/8eeda926f19c43ac85edf6378bfe8504 => generated 0 bytes in 431 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 13 16:07:18.072185 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 902/1779] 10.222.0.50 () {58 vars in 958 bytes} [Tue Sep 13 16:07:17 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 348 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 13 16:07:18.072803 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfdec4ef-78f4-47d8-9d57-fbab89338400 None None] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:07:18.073113 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfdec4ef-78f4-47d8-9d57-fbab89338400 None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:07:18.075728 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfdec4ef-78f4-47d8-9d57-fbab89338400 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:07:18.153546 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-366aa649-ae05-4b77-bcdd-f90e074b70c8 None None] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:07:18.155001 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-366aa649-ae05-4b77-bcdd-f90e074b70c8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:07:18.161111 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.auth.core [None req-bfdec4ef-78f4-47d8-9d57-fbab89338400 None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81438) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:07:18.162483 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-bfdec4ef-78f4-47d8-9d57-fbab89338400 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:07:18.374530 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-366aa649-ae05-4b77-bcdd-f90e074b70c8 None None] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:07:18.506175 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 878/1780] 10.222.0.50 () {58 vars in 958 bytes} [Tue Sep 13 16:07:18 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 446 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:07:18.576129 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f29d47ce-d475-4f90-a9b0-7e5eded0b13a service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f436f07939d4855a21b195bc7566244', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:07:18.577360 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f29d47ce-d475-4f90-a9b0-7e5eded0b13a service neutron] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:07:18.577870 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f29d47ce-d475-4f90-a9b0-7e5eded0b13a service neutron] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:07:18.578384 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f29d47ce-d475-4f90-a9b0-7e5eded0b13a service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:07:18.601139 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f29d47ce-d475-4f90-a9b0-7e5eded0b13a service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:07:18.603002 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfdec4ef-78f4-47d8-9d57-fbab89338400 None None] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:07:18.619681 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f29d47ce-d475-4f90-a9b0-7e5eded0b13a service neutron] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:07:18.623206 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfdec4ef-78f4-47d8-9d57-fbab89338400 None None] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:07:18.668528 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5859dbc3-99fc-482f-9372-8e1642606acf service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f436f07939d4855a21b195bc7566244', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:07:18.673522 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5859dbc3-99fc-482f-9372-8e1642606acf service neutron] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:07:18.673522 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5859dbc3-99fc-482f-9372-8e1642606acf service neutron] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:07:18.673522 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5859dbc3-99fc-482f-9372-8e1642606acf service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:07:18.707578 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 903/1781] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 16:07:18 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 555 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 16:07:18.724965 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5859dbc3-99fc-482f-9372-8e1642606acf service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:07:18.728783 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5859dbc3-99fc-482f-9372-8e1642606acf service neutron] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:07:18.782600 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 879/1782] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 16:07:18 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 181 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 16:07:19.220336 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f29d47ce-d475-4f90-a9b0-7e5eded0b13a service neutron] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:07:19.223223 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-f29d47ce-d475-4f90-a9b0-7e5eded0b13a service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:07:19.249701 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f29d47ce-d475-4f90-a9b0-7e5eded0b13a service neutron] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:07:19.255012 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62b74726-19ec-4d2a-a44f-50fbba6af670 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:07:19.256274 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62b74726-19ec-4d2a-a44f-50fbba6af670 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:07:19.257618 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62b74726-19ec-4d2a-a44f-50fbba6af670 admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:07:19.258176 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62b74726-19ec-4d2a-a44f-50fbba6af670 admin admin] PATH_INFO: `/v3/projects/0f1ad90d23904e05b750d8ba0147119e` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:07:19.273281 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62b74726-19ec-4d2a-a44f-50fbba6af670 admin admin] RBAC: Authorizing `identity:delete_project(project_id=0f1ad90d23904e05b750d8ba0147119e)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:07:19.283663 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62b74726-19ec-4d2a-a44f-50fbba6af670 admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:07:19.346933 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:DELETE=4,keystone:SELECT=95,keystone:INSERT=1 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:07:19.454071 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-62b74726-19ec-4d2a-a44f-50fbba6af670 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '0f1ad90d23904e05b750d8ba0147119e', 'request_id': 'req-62b74726-19ec-4d2a-a44f-50fbba6af670'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:07:19.477073 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-62b74726-19ec-4d2a-a44f-50fbba6af670 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '0f1ad90d23904e05b750d8ba0147119e', 'request_id': 'req-62b74726-19ec-4d2a-a44f-50fbba6af670'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:07:19.486487 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-62b74726-19ec-4d2a-a44f-50fbba6af670 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '0f1ad90d23904e05b750d8ba0147119e', 'request_id': 'req-62b74726-19ec-4d2a-a44f-50fbba6af670'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:07:19.520116 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-62b74726-19ec-4d2a-a44f-50fbba6af670 admin admin] The token cache is being invalidate because project 0f1ad90d23904e05b750d8ba0147119e was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81437) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 13 16:07:19.523231 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-62b74726-19ec-4d2a-a44f-50fbba6af670 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:07:19.527337 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-62b74726-19ec-4d2a-a44f-50fbba6af670 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:07:19.532667 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 904/1783] 10.222.0.50 () {58 vars in 1233 bytes} [Tue Sep 13 16:07:19 2022] DELETE /identity/v3/projects/0f1ad90d23904e05b750d8ba0147119e => generated 0 bytes in 310 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 16:07:19.535760 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4547887-0146-48f1-9252-7a50eb312c36 None None] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:07:19.536133 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4547887-0146-48f1-9252-7a50eb312c36 None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:07:19.536349 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4547887-0146-48f1-9252-7a50eb312c36 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:07:19.576900 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.auth.core [None req-c4547887-0146-48f1-9252-7a50eb312c36 None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81438) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:07:19.578132 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-c4547887-0146-48f1-9252-7a50eb312c36 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:07:19.747433 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-681d38c5-48e6-457c-8871-af3c4d07f692 None None] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:07:19.747433 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-681d38c5-48e6-457c-8871-af3c4d07f692 None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:07:19.747433 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-681d38c5-48e6-457c-8871-af3c4d07f692 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:07:19.764850 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 880/1784] 10.222.0.50 () {58 vars in 958 bytes} [Tue Sep 13 16:07:19 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 230 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:07:19.810874 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.auth.core [None req-681d38c5-48e6-457c-8871-af3c4d07f692 None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81437) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:07:19.816831 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-681d38c5-48e6-457c-8871-af3c4d07f692 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:07:19.865244 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4547887-0146-48f1-9252-7a50eb312c36 None None] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:07:19.876334 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 905/1785] 10.222.0.50 () {58 vars in 958 bytes} [Tue Sep 13 16:07:19 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 137 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:07:19.881133 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4547887-0146-48f1-9252-7a50eb312c36 None None] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:07:19.910617 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd10f344-9bad-49ce-ab48-6a3cb3643829 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:07:19.911370 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd10f344-9bad-49ce-ab48-6a3cb3643829 admin admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:07:19.911652 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd10f344-9bad-49ce-ab48-6a3cb3643829 admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:07:19.918417 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd10f344-9bad-49ce-ab48-6a3cb3643829 admin admin] PATH_INFO: `/v3/domains` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:07:19.919547 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd10f344-9bad-49ce-ab48-6a3cb3643829 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81438) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 16:07:19.919852 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd10f344-9bad-49ce-ab48-6a3cb3643829 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:07:19.929363 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd10f344-9bad-49ce-ab48-6a3cb3643829 admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:07:19.967106 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 881/1786] 10.222.0.50 () {58 vars in 1153 bytes} [Tue Sep 13 16:07:19 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 98 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 16:07:19.968468 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-681d38c5-48e6-457c-8871-af3c4d07f692 None None] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:07:19.981382 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-681d38c5-48e6-457c-8871-af3c4d07f692 None None] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:07:19.994164 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c236d4f-8d6c-404d-bff5-12f1318fdbfb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:07:20.013692 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd10f344-9bad-49ce-ab48-6a3cb3643829 admin admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:07:20.017032 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-dd10f344-9bad-49ce-ab48-6a3cb3643829 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:07:20.034318 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c236d4f-8d6c-404d-bff5-12f1318fdbfb admin admin] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:07:20.035162 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c236d4f-8d6c-404d-bff5-12f1318fdbfb admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:07:20.036061 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c236d4f-8d6c-404d-bff5-12f1318fdbfb admin admin] PATH_INFO: `/v3/projects` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:07:20.040302 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c236d4f-8d6c-404d-bff5-12f1318fdbfb admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:07:20.043412 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c236d4f-8d6c-404d-bff5-12f1318fdbfb admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:07:20.062245 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd10f344-9bad-49ce-ab48-6a3cb3643829 admin admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:07:20.078568 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05153d36-8fa9-4b3c-907d-69b077867267 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:07:20.079993 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05153d36-8fa9-4b3c-907d-69b077867267 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:07:20.080482 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05153d36-8fa9-4b3c-907d-69b077867267 admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:07:20.081847 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05153d36-8fa9-4b3c-907d-69b077867267 admin admin] PATH_INFO: `/v3/projects/b48b1ff8f2904a37af937869f957a8c2` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:07:20.084825 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05153d36-8fa9-4b3c-907d-69b077867267 admin admin] RBAC: Authorizing `identity:delete_project(project_id=b48b1ff8f2904a37af937869f957a8c2)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:07:20.089975 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05153d36-8fa9-4b3c-907d-69b077867267 admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:07:20.137833 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 906/1787] 10.222.0.50 () {60 vars in 1153 bytes} [Tue Sep 13 16:07:19 2022] POST /identity/v3/projects => generated 377 bytes in 170 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 16:07:20.151133 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c236d4f-8d6c-404d-bff5-12f1318fdbfb admin admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:07:20.237237 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=95,keystone:DELETE=5 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:07:20.298563 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-05153d36-8fa9-4b3c-907d-69b077867267 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'b48b1ff8f2904a37af937869f957a8c2', 'request_id': 'req-05153d36-8fa9-4b3c-907d-69b077867267'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:07:20.324856 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-05153d36-8fa9-4b3c-907d-69b077867267 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'b48b1ff8f2904a37af937869f957a8c2', 'request_id': 'req-05153d36-8fa9-4b3c-907d-69b077867267'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:07:20.343651 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-05153d36-8fa9-4b3c-907d-69b077867267 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'b48b1ff8f2904a37af937869f957a8c2', 'request_id': 'req-05153d36-8fa9-4b3c-907d-69b077867267'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:07:20.368906 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-05153d36-8fa9-4b3c-907d-69b077867267 admin admin] The token cache is being invalidate because project b48b1ff8f2904a37af937869f957a8c2 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81438) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 13 16:07:20.369266 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-05153d36-8fa9-4b3c-907d-69b077867267 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:07:20.371672 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c236d4f-8d6c-404d-bff5-12f1318fdbfb admin admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:07:20.372044 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-05153d36-8fa9-4b3c-907d-69b077867267 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:07:20.375359 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 882/1788] 10.222.0.50 () {58 vars in 1233 bytes} [Tue Sep 13 16:07:20 2022] DELETE /identity/v3/projects/b48b1ff8f2904a37af937869f957a8c2 => generated 0 bytes in 362 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 13 16:07:20.378458 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05153d36-8fa9-4b3c-907d-69b077867267 admin admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:07:20.382716 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-05153d36-8fa9-4b3c-907d-69b077867267 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:07:20.523881 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0803ba72-85bd-44dd-9d5c-278cfd276e1c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:07:20.524542 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0803ba72-85bd-44dd-9d5c-278cfd276e1c admin admin] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:07:20.524826 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0803ba72-85bd-44dd-9d5c-278cfd276e1c admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:07:20.525099 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0803ba72-85bd-44dd-9d5c-278cfd276e1c admin admin] PATH_INFO: `/v3/users` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:07:20.525813 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0803ba72-85bd-44dd-9d5c-278cfd276e1c admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:07:20.528634 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0803ba72-85bd-44dd-9d5c-278cfd276e1c admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:07:20.564880 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05153d36-8fa9-4b3c-907d-69b077867267 admin admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:07:20.580077 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 907/1789] 10.222.0.50 () {60 vars in 1144 bytes} [Tue Sep 13 16:07:20 2022] POST /identity/v3/users => generated 289 bytes in 431 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 16:07:20.581021 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0803ba72-85bd-44dd-9d5c-278cfd276e1c admin admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:07:20.584591 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6790cf72-c4cf-47e6-b978-a7a9e9d511dd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:07:20.585917 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6790cf72-c4cf-47e6-b978-a7a9e9d511dd admin admin] REQUEST_METHOD: `DELETE` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:07:20.586430 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6790cf72-c4cf-47e6-b978-a7a9e9d511dd admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:07:20.587044 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6790cf72-c4cf-47e6-b978-a7a9e9d511dd admin admin] PATH_INFO: `/v3/projects/7c2a2507c1314f3ab4fb1b40babae9e5` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:07:20.593793 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6790cf72-c4cf-47e6-b978-a7a9e9d511dd admin admin] RBAC: Authorizing `identity:delete_project(project_id=7c2a2507c1314f3ab4fb1b40babae9e5)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:07:20.595270 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-0803ba72-85bd-44dd-9d5c-278cfd276e1c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:07:20.599010 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6790cf72-c4cf-47e6-b978-a7a9e9d511dd admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:07:20.625643 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0803ba72-85bd-44dd-9d5c-278cfd276e1c admin admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:07:20.640867 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-533f2853-d6eb-4a6f-b017-0af2fa647012 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:07:20.641811 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-533f2853-d6eb-4a6f-b017-0af2fa647012 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:07:20.642317 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-533f2853-d6eb-4a6f-b017-0af2fa647012 admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:07:20.643065 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-533f2853-d6eb-4a6f-b017-0af2fa647012 admin admin] PATH_INFO: `/v3/users/f3d13f6b016648c7ab0d307612b229a6` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:07:20.650912 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-533f2853-d6eb-4a6f-b017-0af2fa647012 admin admin] RBAC: Authorizing `identity:delete_user(user_id=f3d13f6b016648c7ab0d307612b229a6)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:07:20.653203 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-533f2853-d6eb-4a6f-b017-0af2fa647012 admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:07:20.663825 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=88,keystone:DELETE=7,keystone:INSERT=5 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:07:20.725767 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-6790cf72-c4cf-47e6-b978-a7a9e9d511dd admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '7c2a2507c1314f3ab4fb1b40babae9e5', 'request_id': 'req-6790cf72-c4cf-47e6-b978-a7a9e9d511dd'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:07:20.743111 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-6790cf72-c4cf-47e6-b978-a7a9e9d511dd admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '7c2a2507c1314f3ab4fb1b40babae9e5', 'request_id': 'req-6790cf72-c4cf-47e6-b978-a7a9e9d511dd'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:07:20.755873 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-6790cf72-c4cf-47e6-b978-a7a9e9d511dd admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '7c2a2507c1314f3ab4fb1b40babae9e5', 'request_id': 'req-6790cf72-c4cf-47e6-b978-a7a9e9d511dd'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:07:20.770300 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-533f2853-d6eb-4a6f-b017-0af2fa647012 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'f3d13f6b016648c7ab0d307612b229a6'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:07:20.786377 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-533f2853-d6eb-4a6f-b017-0af2fa647012 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'f3d13f6b016648c7ab0d307612b229a6'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:07:20.798651 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-6790cf72-c4cf-47e6-b978-a7a9e9d511dd admin admin] The token cache is being invalidate because project 7c2a2507c1314f3ab4fb1b40babae9e5 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81438) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 13 16:07:20.800709 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-6790cf72-c4cf-47e6-b978-a7a9e9d511dd admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:07:20.801695 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-6790cf72-c4cf-47e6-b978-a7a9e9d511dd admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:07:20.803989 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 883/1790] 10.222.0.50 () {58 vars in 1233 bytes} [Tue Sep 13 16:07:20 2022] DELETE /identity/v3/projects/7c2a2507c1314f3ab4fb1b40babae9e5 => generated 0 bytes in 428 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 16:07:20.805175 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6790cf72-c4cf-47e6-b978-a7a9e9d511dd admin admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:07:20.813020 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-6790cf72-c4cf-47e6-b978-a7a9e9d511dd admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:07:20.824554 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-533f2853-d6eb-4a6f-b017-0af2fa647012 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'f3d13f6b016648c7ab0d307612b229a6'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:07:20.825587 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-533f2853-d6eb-4a6f-b017-0af2fa647012 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'f3d13f6b016648c7ab0d307612b229a6'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:07:20.861914 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-533f2853-d6eb-4a6f-b017-0af2fa647012 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'f3d13f6b016648c7ab0d307612b229a6'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:07:20.875135 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 908/1791] 10.222.0.50 () {58 vars in 1224 bytes} [Tue Sep 13 16:07:20 2022] DELETE /identity/v3/users/f3d13f6b016648c7ab0d307612b229a6 => generated 0 bytes in 294 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 16:07:20.876891 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-533f2853-d6eb-4a6f-b017-0af2fa647012 admin admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:07:20.899713 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-533f2853-d6eb-4a6f-b017-0af2fa647012 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:07:21.075019 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=87,keystone:DELETE=11,keystone:INSERT=2 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:07:21.086078 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6790cf72-c4cf-47e6-b978-a7a9e9d511dd admin admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:07:21.141796 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-533f2853-d6eb-4a6f-b017-0af2fa647012 admin admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:07:21.154273 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f76ee8e-e58e-4807-9b65-69011e845a33 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:07:21.159194 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f76ee8e-e58e-4807-9b65-69011e845a33 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:07:21.159522 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f76ee8e-e58e-4807-9b65-69011e845a33 admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:07:21.159804 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f76ee8e-e58e-4807-9b65-69011e845a33 admin admin] PATH_INFO: `/v3/users/e2b69b7af38d4904ba1ce4a192d423f7` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:07:21.161551 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f76ee8e-e58e-4807-9b65-69011e845a33 admin admin] RBAC: Authorizing `identity:delete_user(user_id=e2b69b7af38d4904ba1ce4a192d423f7)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:07:21.163778 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f76ee8e-e58e-4807-9b65-69011e845a33 admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:07:21.276299 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-2f76ee8e-e58e-4807-9b65-69011e845a33 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'e2b69b7af38d4904ba1ce4a192d423f7'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:07:21.288785 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-691899de-f4be-4085-bddf-09a1b1eabf18 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:07:21.291154 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-691899de-f4be-4085-bddf-09a1b1eabf18 admin admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:07:21.293553 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-691899de-f4be-4085-bddf-09a1b1eabf18 admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:07:21.294178 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-691899de-f4be-4085-bddf-09a1b1eabf18 admin admin] PATH_INFO: `/v3/roles` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:07:21.295562 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-691899de-f4be-4085-bddf-09a1b1eabf18 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:07:21.297373 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-2f76ee8e-e58e-4807-9b65-69011e845a33 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'e2b69b7af38d4904ba1ce4a192d423f7'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:07:21.297797 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-691899de-f4be-4085-bddf-09a1b1eabf18 admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:07:21.311143 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 884/1792] 10.222.0.50 () {58 vars in 1122 bytes} [Tue Sep 13 16:07:20 2022] GET /identity/v3/roles => generated 1420 bytes in 506 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 16:07:21.315195 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-691899de-f4be-4085-bddf-09a1b1eabf18 admin admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:07:21.317694 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-691899de-f4be-4085-bddf-09a1b1eabf18 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:07:21.331866 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-2f76ee8e-e58e-4807-9b65-69011e845a33 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'e2b69b7af38d4904ba1ce4a192d423f7'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:07:21.333825 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-2f76ee8e-e58e-4807-9b65-69011e845a33 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'e2b69b7af38d4904ba1ce4a192d423f7'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:07:21.338019 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-691899de-f4be-4085-bddf-09a1b1eabf18 admin admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:07:21.345130 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fc12d51-3b2a-4766-bcd5-0916b2eb8ade admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:07:21.346005 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fc12d51-3b2a-4766-bcd5-0916b2eb8ade admin admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:07:21.346675 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fc12d51-3b2a-4766-bcd5-0916b2eb8ade admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:07:21.347089 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fc12d51-3b2a-4766-bcd5-0916b2eb8ade admin admin] PATH_INFO: `/v3/roles` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:07:21.347835 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fc12d51-3b2a-4766-bcd5-0916b2eb8ade admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:07:21.349773 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fc12d51-3b2a-4766-bcd5-0916b2eb8ade admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:07:21.352932 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-2f76ee8e-e58e-4807-9b65-69011e845a33 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'e2b69b7af38d4904ba1ce4a192d423f7'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:07:21.371022 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 909/1793] 10.222.0.50 () {58 vars in 1224 bytes} [Tue Sep 13 16:07:20 2022] DELETE /identity/v3/users/e2b69b7af38d4904ba1ce4a192d423f7 => generated 0 bytes in 495 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 16:07:21.389905 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 885/1794] 10.222.0.50 () {58 vars in 1122 bytes} [Tue Sep 13 16:07:21 2022] GET /identity/v3/roles => generated 1420 bytes in 75 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 16:07:21.394464 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f76ee8e-e58e-4807-9b65-69011e845a33 admin admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:07:21.397819 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-2f76ee8e-e58e-4807-9b65-69011e845a33 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:07:21.423543 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=78,keystone:DELETE=20,keystone:INSERT=2 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:07:21.617147 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f76ee8e-e58e-4807-9b65-69011e845a33 admin admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:07:21.625730 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3dc4bc3-43c3-427e-b84b-ca63cea8b54f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:07:21.626850 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3dc4bc3-43c3-427e-b84b-ca63cea8b54f admin admin] REQUEST_METHOD: `PUT` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:07:21.627123 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3dc4bc3-43c3-427e-b84b-ca63cea8b54f admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:07:21.627415 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3dc4bc3-43c3-427e-b84b-ca63cea8b54f admin admin] PATH_INFO: `/v3/projects/8cb3078cf06341a2a6d49dd368db6eb8/users/f43c42e067a74df3ba8f477da7780f1e/roles/15db4aaca3264ae29bde38903ba43471` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:07:21.651860 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3dc4bc3-43c3-427e-b84b-ca63cea8b54f admin admin] RBAC: Authorizing `identity:create_grant(project_id=8cb3078cf06341a2a6d49dd368db6eb8, user_id=f43c42e067a74df3ba8f477da7780f1e, role_id=15db4aaca3264ae29bde38903ba43471)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:07:21.654883 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3dc4bc3-43c3-427e-b84b-ca63cea8b54f admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:07:21.684083 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 910/1795] 10.222.0.50 () {60 vars in 1483 bytes} [Tue Sep 13 16:07:21 2022] PUT /identity/v3/projects/8cb3078cf06341a2a6d49dd368db6eb8/users/f43c42e067a74df3ba8f477da7780f1e/roles/15db4aaca3264ae29bde38903ba43471 => generated 0 bytes in 280 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 16:07:21.688120 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2800ebb3-19dc-48d6-b5b6-c4cf760306f5 None None] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:07:21.688313 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2800ebb3-19dc-48d6-b5b6-c4cf760306f5 None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:07:21.688730 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2800ebb3-19dc-48d6-b5b6-c4cf760306f5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:07:21.748850 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.auth.core [None req-2800ebb3-19dc-48d6-b5b6-c4cf760306f5 None None] MFA Rules not processed for user `f43c42e067a74df3ba8f477da7780f1e`. Rule list: `[]` (Enabled: `True`). {{(pid=81438) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:07:21.752241 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-2800ebb3-19dc-48d6-b5b6-c4cf760306f5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:07:21.774165 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1aebe2a7-75b6-4fa2-977d-f27c18216e02 None None] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:07:21.774463 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1aebe2a7-75b6-4fa2-977d-f27c18216e02 None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:07:21.774798 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1aebe2a7-75b6-4fa2-977d-f27c18216e02 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:07:21.837222 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.auth.core [None req-1aebe2a7-75b6-4fa2-977d-f27c18216e02 None None] MFA Rules not processed for user `2f436f07939d4855a21b195bc7566244`. Rule list: `[]` (Enabled: `True`). {{(pid=81437) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:07:21.840906 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-1aebe2a7-75b6-4fa2-977d-f27c18216e02 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:07:21.955231 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 886/1796] 10.222.0.50 () {58 vars in 958 bytes} [Tue Sep 13 16:07:21 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 268 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:07:21.956132 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2800ebb3-19dc-48d6-b5b6-c4cf760306f5 None None] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:07:21.958952 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-2800ebb3-19dc-48d6-b5b6-c4cf760306f5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:07:22.071560 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 911/1797] 10.222.0.59 () {58 vars in 1015 bytes} [Tue Sep 13 16:07:21 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 298 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:07:22.072432 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1aebe2a7-75b6-4fa2-977d-f27c18216e02 None None] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:07:22.147387 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2800ebb3-19dc-48d6-b5b6-c4cf760306f5 None None] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:07:22.162360 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e28ee19a-6215-43ae-8b9e-70d0fb951ff9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:07:22.163145 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e28ee19a-6215-43ae-8b9e-70d0fb951ff9 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:07:22.174769 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e28ee19a-6215-43ae-8b9e-70d0fb951ff9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:07:22.175602 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e28ee19a-6215-43ae-8b9e-70d0fb951ff9 admin admin] PATH_INFO: `/v3/projects/499a8465673141aba3c5331ceaa17a82` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:07:22.177123 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e28ee19a-6215-43ae-8b9e-70d0fb951ff9 admin admin] RBAC: Authorizing `identity:delete_project(project_id=499a8465673141aba3c5331ceaa17a82)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:07:22.179116 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e28ee19a-6215-43ae-8b9e-70d0fb951ff9 admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:07:22.208437 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:07:22.263862 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1aebe2a7-75b6-4fa2-977d-f27c18216e02 None None] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:07:22.304385 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-e28ee19a-6215-43ae-8b9e-70d0fb951ff9 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '499a8465673141aba3c5331ceaa17a82', 'request_id': 'req-e28ee19a-6215-43ae-8b9e-70d0fb951ff9'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:07:22.326662 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-e28ee19a-6215-43ae-8b9e-70d0fb951ff9 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '499a8465673141aba3c5331ceaa17a82', 'request_id': 'req-e28ee19a-6215-43ae-8b9e-70d0fb951ff9'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:07:22.350401 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-e28ee19a-6215-43ae-8b9e-70d0fb951ff9 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '499a8465673141aba3c5331ceaa17a82', 'request_id': 'req-e28ee19a-6215-43ae-8b9e-70d0fb951ff9'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:07:22.406165 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-e28ee19a-6215-43ae-8b9e-70d0fb951ff9 admin admin] The token cache is being invalidate because project 499a8465673141aba3c5331ceaa17a82 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81438) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 13 16:07:22.407024 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:07:22.407542 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-e28ee19a-6215-43ae-8b9e-70d0fb951ff9 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:07:22.410126 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-e28ee19a-6215-43ae-8b9e-70d0fb951ff9 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:07:22.419025 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 887/1798] 10.222.0.50 () {58 vars in 1233 bytes} [Tue Sep 13 16:07:21 2022] DELETE /identity/v3/projects/499a8465673141aba3c5331ceaa17a82 => generated 0 bytes in 463 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 16:07:22.422708 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e28ee19a-6215-43ae-8b9e-70d0fb951ff9 admin admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:07:22.428065 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-e28ee19a-6215-43ae-8b9e-70d0fb951ff9 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:07:22.472965 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-157e4e99-be0b-4c53-a17f-c680fd96d0d3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:07:22.475394 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-157e4e99-be0b-4c53-a17f-c680fd96d0d3 admin admin] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:07:22.476870 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-157e4e99-be0b-4c53-a17f-c680fd96d0d3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:07:22.477708 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-157e4e99-be0b-4c53-a17f-c680fd96d0d3 admin admin] PATH_INFO: `/v3/projects` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:07:22.480954 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-157e4e99-be0b-4c53-a17f-c680fd96d0d3 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:07:22.489771 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-157e4e99-be0b-4c53-a17f-c680fd96d0d3 admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:07:22.575051 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 912/1799] 10.222.0.50 () {60 vars in 1153 bytes} [Tue Sep 13 16:07:22 2022] POST /identity/v3/projects => generated 377 bytes in 502 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 16:07:22.576333 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-157e4e99-be0b-4c53-a17f-c680fd96d0d3 admin admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:07:22.584886 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-157e4e99-be0b-4c53-a17f-c680fd96d0d3 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:07:22.752985 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e28ee19a-6215-43ae-8b9e-70d0fb951ff9 admin admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:07:22.873790 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-157e4e99-be0b-4c53-a17f-c680fd96d0d3 admin admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:07:22.881360 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fa56712-2d0c-4df6-84ca-0f19b3f36fe9 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f436f07939d4855a21b195bc7566244', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:07:22.882558 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fa56712-2d0c-4df6-84ca-0f19b3f36fe9 service neutron] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:07:22.882558 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fa56712-2d0c-4df6-84ca-0f19b3f36fe9 service neutron] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:07:22.882558 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fa56712-2d0c-4df6-84ca-0f19b3f36fe9 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:07:22.893506 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-5fa56712-2d0c-4df6-84ca-0f19b3f36fe9 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:07:22.955383 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fa56712-2d0c-4df6-84ca-0f19b3f36fe9 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:07:22.967896 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fa56712-2d0c-4df6-84ca-0f19b3f36fe9 service neutron] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:07:23.011544 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 913/1800] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 16:07:22 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 435 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 16:07:23.012675 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df8739c4-d581-4bbb-8ddf-5f4e01777c4f None None] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:07:23.013003 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df8739c4-d581-4bbb-8ddf-5f4e01777c4f None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:07:23.022210 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df8739c4-d581-4bbb-8ddf-5f4e01777c4f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:07:23.049535 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-883a3e56-d14a-4e78-97e7-5cbed274f850 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f436f07939d4855a21b195bc7566244', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:07:23.050090 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-883a3e56-d14a-4e78-97e7-5cbed274f850 service neutron] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:07:23.050388 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-883a3e56-d14a-4e78-97e7-5cbed274f850 service neutron] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:07:23.050670 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-883a3e56-d14a-4e78-97e7-5cbed274f850 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:07:23.081316 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-883a3e56-d14a-4e78-97e7-5cbed274f850 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:07:23.082838 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-883a3e56-d14a-4e78-97e7-5cbed274f850 service neutron] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:07:23.091089 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.auth.core [None req-df8739c4-d581-4bbb-8ddf-5f4e01777c4f None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81437) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:07:23.093239 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-df8739c4-d581-4bbb-8ddf-5f4e01777c4f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:07:23.107333 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 888/1801] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 16:07:22 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 688 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 16:07:23.109516 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-883a3e56-d14a-4e78-97e7-5cbed274f850 service neutron] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:07:23.114614 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-883a3e56-d14a-4e78-97e7-5cbed274f850 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:07:23.371282 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 914/1802] 10.222.0.50 () {58 vars in 958 bytes} [Tue Sep 13 16:07:23 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 359 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:07:23.372281 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df8739c4-d581-4bbb-8ddf-5f4e01777c4f None None] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:07:23.376234 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-df8739c4-d581-4bbb-8ddf-5f4e01777c4f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:07:23.414316 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-883a3e56-d14a-4e78-97e7-5cbed274f850 service neutron] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:07:23.439890 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c148297-1636-45d9-b469-f00dc5e8ad55 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:07:23.442731 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c148297-1636-45d9-b469-f00dc5e8ad55 admin admin] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:07:23.444160 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c148297-1636-45d9-b469-f00dc5e8ad55 admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:07:23.444596 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c148297-1636-45d9-b469-f00dc5e8ad55 admin admin] PATH_INFO: `/v3/users` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:07:23.445398 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c148297-1636-45d9-b469-f00dc5e8ad55 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:07:23.453373 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c148297-1636-45d9-b469-f00dc5e8ad55 admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:07:23.455407 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df8739c4-d581-4bbb-8ddf-5f4e01777c4f None None] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:07:23.496955 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:DELETE=7,keystone:SELECT=89,keystone:INSERT=4 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:07:23.500375 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb5d406f-d5e9-4ec7-bf33-f79b09a8f433 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:07:23.501071 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb5d406f-d5e9-4ec7-bf33-f79b09a8f433 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:07:23.501355 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb5d406f-d5e9-4ec7-bf33-f79b09a8f433 admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:07:23.501632 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb5d406f-d5e9-4ec7-bf33-f79b09a8f433 admin admin] PATH_INFO: `/v3/projects/e46372a21ef248e38fcdf225cd29af15` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:07:23.505675 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 889/1803] 10.222.0.50 () {60 vars in 1144 bytes} [Tue Sep 13 16:07:23 2022] POST /identity/v3/users => generated 289 bytes in 398 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 16:07:23.507700 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b63e02f6-39ef-40a6-8b1e-6cc7b234cc58 None None] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:07:23.508260 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b63e02f6-39ef-40a6-8b1e-6cc7b234cc58 None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:07:23.509039 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b63e02f6-39ef-40a6-8b1e-6cc7b234cc58 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:07:23.513475 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb5d406f-d5e9-4ec7-bf33-f79b09a8f433 admin admin] RBAC: Authorizing `identity:delete_project(project_id=e46372a21ef248e38fcdf225cd29af15)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:07:23.545090 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb5d406f-d5e9-4ec7-bf33-f79b09a8f433 admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:07:23.570060 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.auth.core [None req-b63e02f6-39ef-40a6-8b1e-6cc7b234cc58 None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81438) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:07:23.576555 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-b63e02f6-39ef-40a6-8b1e-6cc7b234cc58 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:07:23.630745 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 890/1804] 10.222.0.50 () {58 vars in 958 bytes} [Tue Sep 13 16:07:23 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 122 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:07:23.632760 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b63e02f6-39ef-40a6-8b1e-6cc7b234cc58 None None] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:07:23.698520 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b63e02f6-39ef-40a6-8b1e-6cc7b234cc58 None None] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:07:23.728783 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a67dbb63-1bd4-4bb0-a7cc-58d46918e0b2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:07:23.740511 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a67dbb63-1bd4-4bb0-a7cc-58d46918e0b2 admin admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:07:23.740803 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a67dbb63-1bd4-4bb0-a7cc-58d46918e0b2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:07:23.741270 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a67dbb63-1bd4-4bb0-a7cc-58d46918e0b2 admin admin] PATH_INFO: `/v3/roles` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:07:23.742274 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a67dbb63-1bd4-4bb0-a7cc-58d46918e0b2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:07:23.746589 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a67dbb63-1bd4-4bb0-a7cc-58d46918e0b2 admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:07:23.805176 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 891/1805] 10.222.0.50 () {58 vars in 1122 bytes} [Tue Sep 13 16:07:23 2022] GET /identity/v3/roles => generated 1420 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 16:07:23.806121 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a67dbb63-1bd4-4bb0-a7cc-58d46918e0b2 admin admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:07:23.845978 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a67dbb63-1bd4-4bb0-a7cc-58d46918e0b2 admin admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:07:23.864738 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-eb5d406f-d5e9-4ec7-bf33-f79b09a8f433 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'e46372a21ef248e38fcdf225cd29af15', 'request_id': 'req-eb5d406f-d5e9-4ec7-bf33-f79b09a8f433'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:07:23.888986 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-eb5d406f-d5e9-4ec7-bf33-f79b09a8f433 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'e46372a21ef248e38fcdf225cd29af15', 'request_id': 'req-eb5d406f-d5e9-4ec7-bf33-f79b09a8f433'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:07:23.921288 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-eb5d406f-d5e9-4ec7-bf33-f79b09a8f433 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'e46372a21ef248e38fcdf225cd29af15', 'request_id': 'req-eb5d406f-d5e9-4ec7-bf33-f79b09a8f433'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:07:23.933485 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=92,keystone:INSERT=2,keystone:DELETE=6 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:07:23.978737 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-eb5d406f-d5e9-4ec7-bf33-f79b09a8f433 admin admin] The token cache is being invalidate because project e46372a21ef248e38fcdf225cd29af15 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81437) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 13 16:07:23.979926 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-eb5d406f-d5e9-4ec7-bf33-f79b09a8f433 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:07:23.982791 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-eb5d406f-d5e9-4ec7-bf33-f79b09a8f433 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:07:23.985835 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 915/1806] 10.222.0.50 () {58 vars in 1233 bytes} [Tue Sep 13 16:07:23 2022] DELETE /identity/v3/projects/e46372a21ef248e38fcdf225cd29af15 => generated 0 bytes in 614 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 16:07:23.987150 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb5d406f-d5e9-4ec7-bf33-f79b09a8f433 admin admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:07:24.011526 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-eb5d406f-d5e9-4ec7-bf33-f79b09a8f433 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:07:24.023718 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e66147f-d51e-4c39-9af9-d12e9521c462 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:07:24.024621 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e66147f-d51e-4c39-9af9-d12e9521c462 admin admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:07:24.024912 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e66147f-d51e-4c39-9af9-d12e9521c462 admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:07:24.027365 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e66147f-d51e-4c39-9af9-d12e9521c462 admin admin] PATH_INFO: `/v3/domains` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:07:24.028065 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e66147f-d51e-4c39-9af9-d12e9521c462 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81438) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 16:07:24.028376 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e66147f-d51e-4c39-9af9-d12e9521c462 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:07:24.030747 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e66147f-d51e-4c39-9af9-d12e9521c462 admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:07:24.080553 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb5d406f-d5e9-4ec7-bf33-f79b09a8f433 admin admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:07:24.082210 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 892/1807] 10.222.0.50 () {58 vars in 1153 bytes} [Tue Sep 13 16:07:23 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 276 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 16:07:24.083568 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e20ce0d-f9b1-4231-9a87-0196e5d9ef74 None None] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:07:24.083855 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e20ce0d-f9b1-4231-9a87-0196e5d9ef74 None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:07:24.084134 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e20ce0d-f9b1-4231-9a87-0196e5d9ef74 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:07:24.094995 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f2b2562-33fc-4af3-b2ee-0bde04c8d32b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:07:24.095716 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f2b2562-33fc-4af3-b2ee-0bde04c8d32b admin admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:07:24.096071 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f2b2562-33fc-4af3-b2ee-0bde04c8d32b admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:07:24.096353 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f2b2562-33fc-4af3-b2ee-0bde04c8d32b admin admin] PATH_INFO: `/v3/roles` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:07:24.100542 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f2b2562-33fc-4af3-b2ee-0bde04c8d32b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:07:24.104797 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f2b2562-33fc-4af3-b2ee-0bde04c8d32b admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:07:24.139336 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 916/1808] 10.222.0.50 () {58 vars in 1122 bytes} [Tue Sep 13 16:07:23 2022] GET /identity/v3/roles => generated 1420 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 16:07:24.141035 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f2b2562-33fc-4af3-b2ee-0bde04c8d32b admin admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:07:24.143675 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.auth.core [None req-6e20ce0d-f9b1-4231-9a87-0196e5d9ef74 None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81438) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:07:24.145098 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-2f2b2562-33fc-4af3-b2ee-0bde04c8d32b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:07:24.146194 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-6e20ce0d-f9b1-4231-9a87-0196e5d9ef74 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:07:24.177511 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 893/1809] 10.222.0.50 () {58 vars in 958 bytes} [Tue Sep 13 16:07:24 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:07:24.180409 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e20ce0d-f9b1-4231-9a87-0196e5d9ef74 None None] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:07:24.188472 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f2b2562-33fc-4af3-b2ee-0bde04c8d32b admin admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:07:24.205610 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cd98d42-f361-423f-8cce-51e4d2a6b66c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:07:24.207867 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cd98d42-f361-423f-8cce-51e4d2a6b66c admin admin] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:07:24.208243 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cd98d42-f361-423f-8cce-51e4d2a6b66c admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:07:24.208994 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cd98d42-f361-423f-8cce-51e4d2a6b66c admin admin] PATH_INFO: `/v3/projects` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:07:24.212290 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1cd98d42-f361-423f-8cce-51e4d2a6b66c admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:07:24.212832 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e20ce0d-f9b1-4231-9a87-0196e5d9ef74 None None] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:07:24.218438 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1cd98d42-f361-423f-8cce-51e4d2a6b66c admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:07:24.226645 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a41702bc-a4c0-4979-be3d-2f776d72a275 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:07:24.227937 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a41702bc-a4c0-4979-be3d-2f776d72a275 admin admin] REQUEST_METHOD: `PUT` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:07:24.229572 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a41702bc-a4c0-4979-be3d-2f776d72a275 admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:07:24.230713 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a41702bc-a4c0-4979-be3d-2f776d72a275 admin admin] PATH_INFO: `/v3/projects/afd9a3784c544737b692986b19f13247/users/c5c76a4638f140739e0d19e303806a9d/roles/15db4aaca3264ae29bde38903ba43471` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:07:24.274115 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a41702bc-a4c0-4979-be3d-2f776d72a275 admin admin] RBAC: Authorizing `identity:create_grant(project_id=afd9a3784c544737b692986b19f13247, user_id=c5c76a4638f140739e0d19e303806a9d, role_id=15db4aaca3264ae29bde38903ba43471)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:07:24.279641 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a41702bc-a4c0-4979-be3d-2f776d72a275 admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:07:24.310674 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 917/1810] 10.222.0.50 () {60 vars in 1153 bytes} [Tue Sep 13 16:07:24 2022] POST /identity/v3/projects => generated 377 bytes in 170 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 16:07:24.318124 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7634cdb2-9b9e-441e-a835-310d51e3f4cb None None] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:07:24.319225 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7634cdb2-9b9e-441e-a835-310d51e3f4cb None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:07:24.319888 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7634cdb2-9b9e-441e-a835-310d51e3f4cb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:07:24.337652 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 894/1811] 10.222.0.50 () {60 vars in 1483 bytes} [Tue Sep 13 16:07:24 2022] PUT /identity/v3/projects/afd9a3784c544737b692986b19f13247/users/c5c76a4638f140739e0d19e303806a9d/roles/15db4aaca3264ae29bde38903ba43471 => generated 0 bytes in 160 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 16:07:24.339504 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a41702bc-a4c0-4979-be3d-2f776d72a275 admin admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:07:24.389270 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.auth.core [None req-7634cdb2-9b9e-441e-a835-310d51e3f4cb None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81437) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:07:24.392734 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-7634cdb2-9b9e-441e-a835-310d51e3f4cb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:07:24.650409 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a41702bc-a4c0-4979-be3d-2f776d72a275 admin admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:07:24.669440 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3e685cf-c6b7-464b-8384-34a36eb7b755 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:07:24.670078 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3e685cf-c6b7-464b-8384-34a36eb7b755 admin admin] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:07:24.670368 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3e685cf-c6b7-464b-8384-34a36eb7b755 admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:07:24.670645 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3e685cf-c6b7-464b-8384-34a36eb7b755 admin admin] PATH_INFO: `/v3/users` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:07:24.671317 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3e685cf-c6b7-464b-8384-34a36eb7b755 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:07:24.673262 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3e685cf-c6b7-464b-8384-34a36eb7b755 admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:07:24.678217 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 918/1812] 10.222.0.50 () {58 vars in 958 bytes} [Tue Sep 13 16:07:24 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 367 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:07:24.679738 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91492932-d916-4b12-a68d-9fe271a6c342 None None] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:07:24.680346 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91492932-d916-4b12-a68d-9fe271a6c342 None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:07:24.680626 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91492932-d916-4b12-a68d-9fe271a6c342 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:07:24.749132 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 895/1813] 10.222.0.50 () {60 vars in 1144 bytes} [Tue Sep 13 16:07:24 2022] POST /identity/v3/users => generated 289 bytes in 409 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 16:07:24.750791 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3e685cf-c6b7-464b-8384-34a36eb7b755 admin admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:07:24.773453 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.auth.core [None req-91492932-d916-4b12-a68d-9fe271a6c342 None None] MFA Rules not processed for user `c5c76a4638f140739e0d19e303806a9d`. Rule list: `[]` (Enabled: `True`). {{(pid=81437) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:07:24.774237 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3e685cf-c6b7-464b-8384-34a36eb7b755 admin admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:07:24.779732 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-91492932-d916-4b12-a68d-9fe271a6c342 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:07:24.806202 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7613f00b-0ebe-48fe-8e6d-92da27e2fe15 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:07:24.807435 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7613f00b-0ebe-48fe-8e6d-92da27e2fe15 admin admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:07:24.808112 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7613f00b-0ebe-48fe-8e6d-92da27e2fe15 admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:07:24.808955 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7613f00b-0ebe-48fe-8e6d-92da27e2fe15 admin admin] PATH_INFO: `/v3/domains` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:07:24.809823 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7613f00b-0ebe-48fe-8e6d-92da27e2fe15 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81438) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 16:07:24.810819 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7613f00b-0ebe-48fe-8e6d-92da27e2fe15 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:07:24.813280 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7613f00b-0ebe-48fe-8e6d-92da27e2fe15 admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:07:24.856473 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 896/1814] 10.222.0.50 () {58 vars in 1153 bytes} [Tue Sep 13 16:07:24 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 106 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 16:07:24.857972 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7613f00b-0ebe-48fe-8e6d-92da27e2fe15 admin admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:07:24.893087 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7613f00b-0ebe-48fe-8e6d-92da27e2fe15 admin admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:07:24.925273 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d63549c-6c0e-46d9-b1a0-ed9ca716559c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:07:24.927213 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d63549c-6c0e-46d9-b1a0-ed9ca716559c admin admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:07:24.928094 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d63549c-6c0e-46d9-b1a0-ed9ca716559c admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:07:24.928582 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d63549c-6c0e-46d9-b1a0-ed9ca716559c admin admin] PATH_INFO: `/v3/roles` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:07:24.929493 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d63549c-6c0e-46d9-b1a0-ed9ca716559c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:07:24.932582 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d63549c-6c0e-46d9-b1a0-ed9ca716559c admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:07:24.967929 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 897/1815] 10.222.0.50 () {58 vars in 1122 bytes} [Tue Sep 13 16:07:24 2022] GET /identity/v3/roles => generated 1420 bytes in 110 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 16:07:24.974872 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d63549c-6c0e-46d9-b1a0-ed9ca716559c admin admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:07:24.995991 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d63549c-6c0e-46d9-b1a0-ed9ca716559c admin admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:07:25.012887 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 919/1816] 10.222.0.50 () {58 vars in 958 bytes} [Tue Sep 13 16:07:24 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 331 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:07:25.016271 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91492932-d916-4b12-a68d-9fe271a6c342 None None] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:07:25.017377 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5147276-3ecd-474b-8d38-e6459891807a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:07:25.021060 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5147276-3ecd-474b-8d38-e6459891807a admin admin] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:07:25.021648 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5147276-3ecd-474b-8d38-e6459891807a admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:07:25.022127 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5147276-3ecd-474b-8d38-e6459891807a admin admin] PATH_INFO: `/v3/projects` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:07:25.023764 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5147276-3ecd-474b-8d38-e6459891807a admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:07:25.028673 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5147276-3ecd-474b-8d38-e6459891807a admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:07:25.046269 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91492932-d916-4b12-a68d-9fe271a6c342 None None] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:07:25.048028 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:07:25.068221 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73156269-912a-4066-8543-56a2bde372cd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:07:25.069829 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73156269-912a-4066-8543-56a2bde372cd admin admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:07:25.070316 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73156269-912a-4066-8543-56a2bde372cd admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:07:25.070749 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73156269-912a-4066-8543-56a2bde372cd admin admin] PATH_INFO: `/v3/roles` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:07:25.071437 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73156269-912a-4066-8543-56a2bde372cd admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:07:25.073699 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73156269-912a-4066-8543-56a2bde372cd admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:07:25.087143 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 920/1817] 10.222.0.50 () {58 vars in 1122 bytes} [Tue Sep 13 16:07:25 2022] GET /identity/v3/roles => generated 1420 bytes in 73 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 16:07:25.102880 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73156269-912a-4066-8543-56a2bde372cd admin admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:07:25.113654 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 898/1818] 10.222.0.50 () {60 vars in 1153 bytes} [Tue Sep 13 16:07:24 2022] POST /identity/v3/projects => generated 381 bytes in 139 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 16:07:25.113654 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5147276-3ecd-474b-8d38-e6459891807a admin admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:07:25.155677 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73156269-912a-4066-8543-56a2bde372cd admin admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:07:25.214610 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5147276-3ecd-474b-8d38-e6459891807a admin admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:07:25.220431 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e495c5fc-5de9-4b45-89cc-c26812362463 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:07:25.221132 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e495c5fc-5de9-4b45-89cc-c26812362463 admin admin] REQUEST_METHOD: `PUT` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:07:25.221492 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e495c5fc-5de9-4b45-89cc-c26812362463 admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:07:25.221778 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e495c5fc-5de9-4b45-89cc-c26812362463 admin admin] PATH_INFO: `/v3/projects/fe981edab10d46d5a3865fef51134564/users/5887f4924c064e76bd9a8b2d59432aa6/roles/15db4aaca3264ae29bde38903ba43471` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:07:25.243114 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e495c5fc-5de9-4b45-89cc-c26812362463 admin admin] RBAC: Authorizing `identity:create_grant(project_id=fe981edab10d46d5a3865fef51134564, user_id=5887f4924c064e76bd9a8b2d59432aa6, role_id=15db4aaca3264ae29bde38903ba43471)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:07:25.246562 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e495c5fc-5de9-4b45-89cc-c26812362463 admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:07:25.261841 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:DELETE=1,keystone:SELECT=98,keystone:INSERT=1 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:07:25.266590 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 899/1819] 10.222.0.50 () {60 vars in 1483 bytes} [Tue Sep 13 16:07:25 2022] PUT /identity/v3/projects/fe981edab10d46d5a3865fef51134564/users/5887f4924c064e76bd9a8b2d59432aa6/roles/15db4aaca3264ae29bde38903ba43471 => generated 0 bytes in 155 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 16:07:25.269095 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e495c5fc-5de9-4b45-89cc-c26812362463 admin admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:07:25.315241 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63ae37e1-f164-432e-b969-4fb12554e910 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:07:25.316600 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63ae37e1-f164-432e-b969-4fb12554e910 admin admin] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:07:25.316963 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63ae37e1-f164-432e-b969-4fb12554e910 admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:07:25.317159 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63ae37e1-f164-432e-b969-4fb12554e910 admin admin] PATH_INFO: `/v3/projects` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:07:25.320274 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63ae37e1-f164-432e-b969-4fb12554e910 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:07:25.328111 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63ae37e1-f164-432e-b969-4fb12554e910 admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:07:25.396992 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 921/1820] 10.222.0.50 () {60 vars in 1153 bytes} [Tue Sep 13 16:07:25 2022] POST /identity/v3/projects => generated 377 bytes in 309 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 16:07:25.398691 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1debb501-6ab1-4f2c-b353-c43355009a1e None None] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:07:25.399117 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1debb501-6ab1-4f2c-b353-c43355009a1e None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:07:25.399577 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1debb501-6ab1-4f2c-b353-c43355009a1e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:07:25.437855 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e495c5fc-5de9-4b45-89cc-c26812362463 admin admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:07:25.457692 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.auth.core [None req-1debb501-6ab1-4f2c-b353-c43355009a1e None None] MFA Rules not processed for user `5887f4924c064e76bd9a8b2d59432aa6`. Rule list: `[]` (Enabled: `True`). {{(pid=81437) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:07:25.459416 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-1debb501-6ab1-4f2c-b353-c43355009a1e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:07:25.564208 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee6dc4e4-f337-486e-a142-a2113e5161d4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:07:25.565259 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee6dc4e4-f337-486e-a142-a2113e5161d4 admin admin] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:07:25.565866 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee6dc4e4-f337-486e-a142-a2113e5161d4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:07:25.566222 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee6dc4e4-f337-486e-a142-a2113e5161d4 admin admin] PATH_INFO: `/v3/users` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:07:25.567213 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee6dc4e4-f337-486e-a142-a2113e5161d4 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:07:25.569826 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee6dc4e4-f337-486e-a142-a2113e5161d4 admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:07:25.601613 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 900/1821] 10.222.0.50 () {60 vars in 1144 bytes} [Tue Sep 13 16:07:25 2022] POST /identity/v3/users => generated 291 bytes in 334 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 16:07:25.602877 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee6dc4e4-f337-486e-a142-a2113e5161d4 admin admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:07:25.625513 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee6dc4e4-f337-486e-a142-a2113e5161d4 admin admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:07:25.634684 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cda00cd7-4182-468f-83ae-28aa18f8ad5a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:07:25.637177 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cda00cd7-4182-468f-83ae-28aa18f8ad5a admin admin] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:07:25.637819 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cda00cd7-4182-468f-83ae-28aa18f8ad5a admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:07:25.638777 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cda00cd7-4182-468f-83ae-28aa18f8ad5a admin admin] PATH_INFO: `/v3/users` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:07:25.640141 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cda00cd7-4182-468f-83ae-28aa18f8ad5a admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:07:25.645075 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cda00cd7-4182-468f-83ae-28aa18f8ad5a admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:07:25.676293 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:07:25.704756 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 901/1822] 10.222.0.50 () {60 vars in 1144 bytes} [Tue Sep 13 16:07:25 2022] POST /identity/v3/users => generated 295 bytes in 102 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 16:07:25.707782 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cda00cd7-4182-468f-83ae-28aa18f8ad5a admin admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:07:25.727414 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 922/1823] 10.222.0.50 () {58 vars in 958 bytes} [Tue Sep 13 16:07:25 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 329 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:07:25.728362 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1debb501-6ab1-4f2c-b353-c43355009a1e None None] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:07:25.734260 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cda00cd7-4182-468f-83ae-28aa18f8ad5a admin admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:07:25.740374 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b00f002a-97e8-44ad-8755-6b968ec75f33 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:07:25.741175 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b00f002a-97e8-44ad-8755-6b968ec75f33 admin admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:07:25.741598 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b00f002a-97e8-44ad-8755-6b968ec75f33 admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:07:25.742053 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b00f002a-97e8-44ad-8755-6b968ec75f33 admin admin] PATH_INFO: `/v3/roles` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:07:25.742973 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b00f002a-97e8-44ad-8755-6b968ec75f33 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:07:25.744878 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b00f002a-97e8-44ad-8755-6b968ec75f33 admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:07:25.763689 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 902/1824] 10.222.0.50 () {58 vars in 1122 bytes} [Tue Sep 13 16:07:25 2022] GET /identity/v3/roles => generated 1420 bytes in 59 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 16:07:25.764923 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b00f002a-97e8-44ad-8755-6b968ec75f33 admin admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:07:25.777436 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1debb501-6ab1-4f2c-b353-c43355009a1e None None] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:07:25.784837 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b00f002a-97e8-44ad-8755-6b968ec75f33 admin admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:07:25.787800 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06bfd069-f12b-4bb6-894a-683f9ff1d9d0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:07:25.788585 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06bfd069-f12b-4bb6-894a-683f9ff1d9d0 admin admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:07:25.788871 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06bfd069-f12b-4bb6-894a-683f9ff1d9d0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:07:25.789149 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06bfd069-f12b-4bb6-894a-683f9ff1d9d0 admin admin] PATH_INFO: `/v3/roles` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:07:25.792082 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06bfd069-f12b-4bb6-894a-683f9ff1d9d0 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:07:25.792794 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-864eebc0-7d4b-4095-9104-57863b6231e3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:07:25.793666 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-864eebc0-7d4b-4095-9104-57863b6231e3 admin admin] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:07:25.793844 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06bfd069-f12b-4bb6-894a-683f9ff1d9d0 admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:07:25.794333 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-864eebc0-7d4b-4095-9104-57863b6231e3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:07:25.794803 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-864eebc0-7d4b-4095-9104-57863b6231e3 admin admin] PATH_INFO: `/v3/projects` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:07:25.795679 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-864eebc0-7d4b-4095-9104-57863b6231e3 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:07:25.798037 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-864eebc0-7d4b-4095-9104-57863b6231e3 admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:07:25.829314 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 923/1825] 10.222.0.50 () {58 vars in 1122 bytes} [Tue Sep 13 16:07:25 2022] GET /identity/v3/roles => generated 1420 bytes in 95 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 16:07:25.830202 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06bfd069-f12b-4bb6-894a-683f9ff1d9d0 admin admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:07:25.837193 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 903/1826] 10.222.0.50 () {60 vars in 1153 bytes} [Tue Sep 13 16:07:25 2022] POST /identity/v3/projects => generated 377 bytes in 72 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 16:07:25.839151 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-864eebc0-7d4b-4095-9104-57863b6231e3 admin admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:07:25.855703 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06bfd069-f12b-4bb6-894a-683f9ff1d9d0 admin admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:07:25.984628 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-864eebc0-7d4b-4095-9104-57863b6231e3 admin admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:07:25.990156 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb78ebe1-6211-4f04-9f46-88524b6e3853 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:07:25.991013 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb78ebe1-6211-4f04-9f46-88524b6e3853 admin admin] REQUEST_METHOD: `PUT` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:07:25.991365 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb78ebe1-6211-4f04-9f46-88524b6e3853 admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:07:25.991829 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb78ebe1-6211-4f04-9f46-88524b6e3853 admin admin] PATH_INFO: `/v3/projects/4b57870b0d4e416f8ea06c6a1683a10f/users/e20033f63de244b4a3e8bf97bfb1404a/roles/031d4e1a23e34f71858fc4a6dfbd4ccb` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:07:26.044724 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1aa45f6d-34e5-4bfa-922d-b5c5e273b61a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:07:26.045375 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1aa45f6d-34e5-4bfa-922d-b5c5e273b61a admin admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:07:26.045629 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1aa45f6d-34e5-4bfa-922d-b5c5e273b61a admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:07:26.045902 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1aa45f6d-34e5-4bfa-922d-b5c5e273b61a admin admin] PATH_INFO: `/v3/roles` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:07:26.046556 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1aa45f6d-34e5-4bfa-922d-b5c5e273b61a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:07:26.049775 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1aa45f6d-34e5-4bfa-922d-b5c5e273b61a admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:07:26.054534 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb78ebe1-6211-4f04-9f46-88524b6e3853 admin admin] RBAC: Authorizing `identity:create_grant(project_id=4b57870b0d4e416f8ea06c6a1683a10f, user_id=e20033f63de244b4a3e8bf97bfb1404a, role_id=031d4e1a23e34f71858fc4a6dfbd4ccb)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:07:26.067664 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb78ebe1-6211-4f04-9f46-88524b6e3853 admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:07:26.069603 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 924/1827] 10.222.0.50 () {58 vars in 1122 bytes} [Tue Sep 13 16:07:25 2022] GET /identity/v3/roles => generated 1420 bytes in 239 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 16:07:26.070470 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1aa45f6d-34e5-4bfa-922d-b5c5e273b61a admin admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:07:26.097490 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1aa45f6d-34e5-4bfa-922d-b5c5e273b61a admin admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:07:26.103447 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 904/1828] 10.222.0.50 () {60 vars in 1483 bytes} [Tue Sep 13 16:07:25 2022] PUT /identity/v3/projects/4b57870b0d4e416f8ea06c6a1683a10f/users/e20033f63de244b4a3e8bf97bfb1404a/roles/031d4e1a23e34f71858fc4a6dfbd4ccb => generated 0 bytes in 265 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 16:07:26.110428 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8ee74a8-4552-4f16-b606-e1a6fc7ae86b None None] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:07:26.111065 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8ee74a8-4552-4f16-b606-e1a6fc7ae86b None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:07:26.111387 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8ee74a8-4552-4f16-b606-e1a6fc7ae86b None None] PATH_INFO: `/` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:07:26.115208 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 905/1829] 10.222.0.50 () {54 vars in 921 bytes} [Tue Sep 13 16:07:26 2022] GET /identity => generated 270 bytes in 11 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 16:07:26.118187 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8ee74a8-4552-4f16-b606-e1a6fc7ae86b None None] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:07:26.241864 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:07:26.282925 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8ee74a8-4552-4f16-b606-e1a6fc7ae86b None None] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:07:26.311072 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f67b0f9-53f7-4cf3-8181-d8d9517d8df7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:07:26.311072 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f67b0f9-53f7-4cf3-8181-d8d9517d8df7 admin admin] REQUEST_METHOD: `PUT` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:07:26.311072 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f67b0f9-53f7-4cf3-8181-d8d9517d8df7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:07:26.311072 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f67b0f9-53f7-4cf3-8181-d8d9517d8df7 admin admin] PATH_INFO: `/v3/projects/d0c3c872b5d74321b4e09f8d170a2207/users/f804092800c54ba0b1addcde7000012c/roles/15db4aaca3264ae29bde38903ba43471` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:07:26.341754 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a9410fc-6c5b-43bb-8ccf-ea33fa6b5bd1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:07:26.341945 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f67b0f9-53f7-4cf3-8181-d8d9517d8df7 admin admin] RBAC: Authorizing `identity:create_grant(project_id=d0c3c872b5d74321b4e09f8d170a2207, user_id=f804092800c54ba0b1addcde7000012c, role_id=15db4aaca3264ae29bde38903ba43471)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:07:26.342557 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a9410fc-6c5b-43bb-8ccf-ea33fa6b5bd1 admin admin] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:07:26.342672 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a9410fc-6c5b-43bb-8ccf-ea33fa6b5bd1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:07:26.342931 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a9410fc-6c5b-43bb-8ccf-ea33fa6b5bd1 admin admin] PATH_INFO: `/v3/users` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:07:26.346074 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a9410fc-6c5b-43bb-8ccf-ea33fa6b5bd1 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:07:26.346824 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f67b0f9-53f7-4cf3-8181-d8d9517d8df7 admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:07:26.354274 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a9410fc-6c5b-43bb-8ccf-ea33fa6b5bd1 admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:07:26.383053 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 906/1830] 10.222.0.50 () {60 vars in 1483 bytes} [Tue Sep 13 16:07:26 2022] PUT /identity/v3/projects/d0c3c872b5d74321b4e09f8d170a2207/users/f804092800c54ba0b1addcde7000012c/roles/15db4aaca3264ae29bde38903ba43471 => generated 0 bytes in 259 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 16:07:26.383053 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f67b0f9-53f7-4cf3-8181-d8d9517d8df7 admin admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:07:26.418612 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 925/1831] 10.222.0.50 () {60 vars in 1144 bytes} [Tue Sep 13 16:07:26 2022] POST /identity/v3/users => generated 295 bytes in 348 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 16:07:26.423842 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0a1c5f0-6a96-4e24-95c8-562a5732e579 None None] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:07:26.430410 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0a1c5f0-6a96-4e24-95c8-562a5732e579 None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:07:26.431307 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0a1c5f0-6a96-4e24-95c8-562a5732e579 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:07:26.510974 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.auth.core [None req-e0a1c5f0-6a96-4e24-95c8-562a5732e579 None None] MFA Rules not processed for user `4e039b46c5e3445bbb39904b00c5430e`. Rule list: `[]` (Enabled: `True`). {{(pid=81437) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:07:26.514013 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-e0a1c5f0-6a96-4e24-95c8-562a5732e579 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:07:26.530152 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:INSERT=6,keystone:SELECT=94 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:07:26.706563 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f67b0f9-53f7-4cf3-8181-d8d9517d8df7 admin admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:07:26.724670 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df9bc8a4-4b10-4b73-98f3-e7ec0cc2bda3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:07:26.725426 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df9bc8a4-4b10-4b73-98f3-e7ec0cc2bda3 admin admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:07:26.725763 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df9bc8a4-4b10-4b73-98f3-e7ec0cc2bda3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:07:26.726035 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df9bc8a4-4b10-4b73-98f3-e7ec0cc2bda3 admin admin] PATH_INFO: `/v3/roles` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:07:26.726736 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df9bc8a4-4b10-4b73-98f3-e7ec0cc2bda3 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:07:26.730695 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 926/1832] 10.222.0.50 () {58 vars in 1018 bytes} [Tue Sep 13 16:07:26 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 311 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:07:26.731398 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df9bc8a4-4b10-4b73-98f3-e7ec0cc2bda3 admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:07:26.732214 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5538fe98-6098-48fd-81b7-f2a3bf9504f1 None None] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:07:26.732506 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5538fe98-6098-48fd-81b7-f2a3bf9504f1 None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:07:26.738788 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5538fe98-6098-48fd-81b7-f2a3bf9504f1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:07:26.765227 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 907/1833] 10.222.0.50 () {58 vars in 1122 bytes} [Tue Sep 13 16:07:26 2022] GET /identity/v3/roles => generated 1420 bytes in 390 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 16:07:26.769561 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df9bc8a4-4b10-4b73-98f3-e7ec0cc2bda3 admin admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:07:26.808608 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df9bc8a4-4b10-4b73-98f3-e7ec0cc2bda3 admin admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:07:26.825491 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-526ef6ae-a826-4357-b0fb-d0c6d76b13a1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:07:26.826624 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-526ef6ae-a826-4357-b0fb-d0c6d76b13a1 admin admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:07:26.827357 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-526ef6ae-a826-4357-b0fb-d0c6d76b13a1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:07:26.827648 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-526ef6ae-a826-4357-b0fb-d0c6d76b13a1 admin admin] PATH_INFO: `/v3/roles` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:07:26.828300 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-526ef6ae-a826-4357-b0fb-d0c6d76b13a1 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:07:26.831933 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-526ef6ae-a826-4357-b0fb-d0c6d76b13a1 admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:07:26.850957 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.auth.core [None req-5538fe98-6098-48fd-81b7-f2a3bf9504f1 None None] MFA Rules not processed for user `f804092800c54ba0b1addcde7000012c`. Rule list: `[]` (Enabled: `True`). {{(pid=81437) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:07:26.850957 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-5538fe98-6098-48fd-81b7-f2a3bf9504f1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:07:26.879883 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 908/1834] 10.222.0.50 () {58 vars in 1122 bytes} [Tue Sep 13 16:07:26 2022] GET /identity/v3/roles => generated 1420 bytes in 111 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 16:07:26.879883 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-526ef6ae-a826-4357-b0fb-d0c6d76b13a1 admin admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:07:26.920576 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-526ef6ae-a826-4357-b0fb-d0c6d76b13a1 admin admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:07:26.948337 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bec92d93-c3c9-4f9a-992d-8462fbf7b605 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:07:26.949034 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bec92d93-c3c9-4f9a-992d-8462fbf7b605 admin admin] REQUEST_METHOD: `PUT` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:07:26.949318 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bec92d93-c3c9-4f9a-992d-8462fbf7b605 admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:07:26.949596 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bec92d93-c3c9-4f9a-992d-8462fbf7b605 admin admin] PATH_INFO: `/v3/projects/4b57870b0d4e416f8ea06c6a1683a10f/users/e20033f63de244b4a3e8bf97bfb1404a/roles/031d4e1a23e34f71858fc4a6dfbd4ccb` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:07:26.953190 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bec92d93-c3c9-4f9a-992d-8462fbf7b605 admin admin] RBAC: Authorizing `identity:create_grant(project_id=4b57870b0d4e416f8ea06c6a1683a10f, user_id=e20033f63de244b4a3e8bf97bfb1404a, role_id=031d4e1a23e34f71858fc4a6dfbd4ccb)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:07:26.960361 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bec92d93-c3c9-4f9a-992d-8462fbf7b605 admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:07:27.016374 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 909/1835] 10.222.0.50 () {60 vars in 1483 bytes} [Tue Sep 13 16:07:26 2022] PUT /identity/v3/projects/4b57870b0d4e416f8ea06c6a1683a10f/users/e20033f63de244b4a3e8bf97bfb1404a/roles/031d4e1a23e34f71858fc4a6dfbd4ccb => generated 0 bytes in 138 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 16:07:27.017727 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bec92d93-c3c9-4f9a-992d-8462fbf7b605 admin admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:07:27.023099 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-bec92d93-c3c9-4f9a-992d-8462fbf7b605 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:07:27.043452 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 927/1836] 10.222.0.50 () {58 vars in 958 bytes} [Tue Sep 13 16:07:26 2022] POST /identity/v3/auth/tokens => generated 3258 bytes in 312 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:07:27.048011 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5538fe98-6098-48fd-81b7-f2a3bf9504f1 None None] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:07:27.089272 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5538fe98-6098-48fd-81b7-f2a3bf9504f1 None None] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:07:27.279392 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:07:27.328263 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af2e35ee-6173-4a48-8e6f-aba402e282ac admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:07:27.332694 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af2e35ee-6173-4a48-8e6f-aba402e282ac admin admin] REQUEST_METHOD: `PUT` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:07:27.334714 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af2e35ee-6173-4a48-8e6f-aba402e282ac admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:07:27.336429 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af2e35ee-6173-4a48-8e6f-aba402e282ac admin admin] PATH_INFO: `/v3/projects/06471f253f8040628a6bb747e757c7e5/users/6c569801e5e34625b8ed11d688541dab/roles/031d4e1a23e34f71858fc4a6dfbd4ccb` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:07:27.373340 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bec92d93-c3c9-4f9a-992d-8462fbf7b605 admin admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:07:27.392195 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cadfaae0-998a-4286-b630-60ed1029a4b8 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4e039b46c5e3445bbb39904b00c5430e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'admin', 'ResellerAdmin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:07:27.393304 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cadfaae0-998a-4286-b630-60ed1029a4b8 service nova] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:07:27.394327 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cadfaae0-998a-4286-b630-60ed1029a4b8 service nova] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:07:27.395871 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cadfaae0-998a-4286-b630-60ed1029a4b8 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:07:27.408462 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af2e35ee-6173-4a48-8e6f-aba402e282ac admin admin] RBAC: Authorizing `identity:create_grant(project_id=06471f253f8040628a6bb747e757c7e5, user_id=6c569801e5e34625b8ed11d688541dab, role_id=031d4e1a23e34f71858fc4a6dfbd4ccb)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:07:27.415194 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af2e35ee-6173-4a48-8e6f-aba402e282ac admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:07:27.443315 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 928/1837] 10.222.0.50 () {60 vars in 1483 bytes} [Tue Sep 13 16:07:27 2022] PUT /identity/v3/projects/06471f253f8040628a6bb747e757c7e5/users/6c569801e5e34625b8ed11d688541dab/roles/031d4e1a23e34f71858fc4a6dfbd4ccb => generated 0 bytes in 399 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 16:07:27.444070 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cadfaae0-998a-4286-b630-60ed1029a4b8 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:07:27.444911 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b917a38-c774-4b78-b213-26f42482bdf7 None None] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:07:27.445293 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b917a38-c774-4b78-b213-26f42482bdf7 None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:07:27.446034 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b917a38-c774-4b78-b213-26f42482bdf7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:07:27.459182 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cadfaae0-998a-4286-b630-60ed1029a4b8 service nova] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:07:27.480682 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.auth.core [None req-7b917a38-c774-4b78-b213-26f42482bdf7 None None] MFA Rules not processed for user `e20033f63de244b4a3e8bf97bfb1404a`. Rule list: `[]` (Enabled: `True`). {{(pid=81437) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:07:27.482381 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-7b917a38-c774-4b78-b213-26f42482bdf7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:07:27.503078 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 910/1838] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 16:07:27 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 479 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 16:07:27.503078 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cadfaae0-998a-4286-b630-60ed1029a4b8 service nova] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:07:27.519080 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cadfaae0-998a-4286-b630-60ed1029a4b8 service nova] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:07:27.666402 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02d9ce3e-745e-4f1b-9d89-bb3d118c70fb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:07:27.667267 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02d9ce3e-745e-4f1b-9d89-bb3d118c70fb admin admin] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:07:27.667614 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02d9ce3e-745e-4f1b-9d89-bb3d118c70fb admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:07:27.667896 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02d9ce3e-745e-4f1b-9d89-bb3d118c70fb admin admin] PATH_INFO: `/v3/projects` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:07:27.668550 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02d9ce3e-745e-4f1b-9d89-bb3d118c70fb admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:07:27.670782 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02d9ce3e-745e-4f1b-9d89-bb3d118c70fb admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:07:27.700941 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 929/1839] 10.222.0.50 () {58 vars in 958 bytes} [Tue Sep 13 16:07:27 2022] POST /identity/v3/auth/tokens => generated 3321 bytes in 257 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:07:27.702226 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b917a38-c774-4b78-b213-26f42482bdf7 None None] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:07:27.715663 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b917a38-c774-4b78-b213-26f42482bdf7 None None] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:07:27.733492 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b4be99b-971f-41a6-b7d9-4cbfebe328e6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:07:27.734136 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b4be99b-971f-41a6-b7d9-4cbfebe328e6 admin admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:07:27.734419 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b4be99b-971f-41a6-b7d9-4cbfebe328e6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:07:27.734700 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b4be99b-971f-41a6-b7d9-4cbfebe328e6 admin admin] PATH_INFO: `/v3/roles` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:07:27.735395 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b4be99b-971f-41a6-b7d9-4cbfebe328e6 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:07:27.741723 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b4be99b-971f-41a6-b7d9-4cbfebe328e6 admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:07:27.764159 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 911/1840] 10.222.0.50 () {60 vars in 1153 bytes} [Tue Sep 13 16:07:27 2022] POST /identity/v3/projects => generated 379 bytes in 267 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 16:07:27.769984 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02d9ce3e-745e-4f1b-9d89-bb3d118c70fb admin admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:07:27.774161 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-02d9ce3e-745e-4f1b-9d89-bb3d118c70fb admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:07:27.779827 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 930/1841] 10.222.0.50 () {58 vars in 1122 bytes} [Tue Sep 13 16:07:27 2022] GET /identity/v3/roles => generated 1420 bytes in 78 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 16:07:27.781849 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b4be99b-971f-41a6-b7d9-4cbfebe328e6 admin admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:07:27.794299 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:07:27.830385 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b4be99b-971f-41a6-b7d9-4cbfebe328e6 admin admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:07:27.915137 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02d9ce3e-745e-4f1b-9d89-bb3d118c70fb admin admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:07:27.923804 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0366660a-94b0-4403-85cc-af3d14260cb8 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7fd6b5afd67b48f9b0f0597ccd4cf4df', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:07:27.943483 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0366660a-94b0-4403-85cc-af3d14260cb8 service cinder] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:07:27.944008 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0366660a-94b0-4403-85cc-af3d14260cb8 service cinder] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:07:27.944366 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0366660a-94b0-4403-85cc-af3d14260cb8 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:07:27.947880 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-0366660a-94b0-4403-85cc-af3d14260cb8 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:07:27.987525 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26db7c4f-f472-4a88-a986-d9475215c216 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:07:27.988161 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26db7c4f-f472-4a88-a986-d9475215c216 admin admin] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:07:27.988431 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26db7c4f-f472-4a88-a986-d9475215c216 admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:07:27.988707 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26db7c4f-f472-4a88-a986-d9475215c216 admin admin] PATH_INFO: `/v3/users` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:07:27.989344 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26db7c4f-f472-4a88-a986-d9475215c216 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:07:27.991319 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26db7c4f-f472-4a88-a986-d9475215c216 admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:07:28.036441 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 931/1842] 10.222.0.50 () {60 vars in 1144 bytes} [Tue Sep 13 16:07:27 2022] POST /identity/v3/users => generated 296 bytes in 256 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 16:07:28.037367 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26db7c4f-f472-4a88-a986-d9475215c216 admin admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:07:28.071619 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26db7c4f-f472-4a88-a986-d9475215c216 admin admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:07:28.078358 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc442d10-3ed3-4209-ac1c-5a40d6e518b5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:07:28.078358 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc442d10-3ed3-4209-ac1c-5a40d6e518b5 admin admin] REQUEST_METHOD: `PUT` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:07:28.078358 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc442d10-3ed3-4209-ac1c-5a40d6e518b5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:07:28.078358 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc442d10-3ed3-4209-ac1c-5a40d6e518b5 admin admin] PATH_INFO: `/v3/projects/06471f253f8040628a6bb747e757c7e5/users/6c569801e5e34625b8ed11d688541dab/roles/031d4e1a23e34f71858fc4a6dfbd4ccb` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:07:28.083519 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc442d10-3ed3-4209-ac1c-5a40d6e518b5 admin admin] RBAC: Authorizing `identity:create_grant(project_id=06471f253f8040628a6bb747e757c7e5, user_id=6c569801e5e34625b8ed11d688541dab, role_id=031d4e1a23e34f71858fc4a6dfbd4ccb)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:07:28.088080 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc442d10-3ed3-4209-ac1c-5a40d6e518b5 admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:07:28.124868 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0366660a-94b0-4403-85cc-af3d14260cb8 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:07:28.125117 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 932/1843] 10.222.0.50 () {60 vars in 1483 bytes} [Tue Sep 13 16:07:28 2022] PUT /identity/v3/projects/06471f253f8040628a6bb747e757c7e5/users/6c569801e5e34625b8ed11d688541dab/roles/031d4e1a23e34f71858fc4a6dfbd4ccb => generated 0 bytes in 88 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 16:07:28.126859 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0366660a-94b0-4403-85cc-af3d14260cb8 service cinder] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:07:28.127115 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dda9ab60-2831-4e7a-a8ef-363251b6fe89 None None] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:07:28.127410 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dda9ab60-2831-4e7a-a8ef-363251b6fe89 None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:07:28.129648 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dda9ab60-2831-4e7a-a8ef-363251b6fe89 None None] PATH_INFO: `/` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:07:28.137937 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 933/1844] 10.222.0.50 () {54 vars in 921 bytes} [Tue Sep 13 16:07:28 2022] GET /identity => generated 270 bytes in 12 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 16:07:28.138763 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dda9ab60-2831-4e7a-a8ef-363251b6fe89 None None] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:07:28.153271 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 912/1845] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 16:07:27 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 384 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 16:07:28.155445 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8491d1da-1666-4498-b492-9ac18631c1cb None None] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:07:28.155794 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8491d1da-1666-4498-b492-9ac18631c1cb None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:07:28.156104 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8491d1da-1666-4498-b492-9ac18631c1cb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:07:28.214706 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:07:28.230019 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.auth.core [None req-8491d1da-1666-4498-b492-9ac18631c1cb None None] MFA Rules not processed for user `6c569801e5e34625b8ed11d688541dab`. Rule list: `[]` (Enabled: `True`). {{(pid=81438) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:07:28.233648 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-8491d1da-1666-4498-b492-9ac18631c1cb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:07:28.325181 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dda9ab60-2831-4e7a-a8ef-363251b6fe89 None None] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:07:28.330975 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43ae6f71-5178-42f9-b970-5520f85a060f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:07:28.332192 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43ae6f71-5178-42f9-b970-5520f85a060f admin admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:07:28.332623 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43ae6f71-5178-42f9-b970-5520f85a060f admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:07:28.333045 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43ae6f71-5178-42f9-b970-5520f85a060f admin admin] PATH_INFO: `/v3/roles` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:07:28.334710 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43ae6f71-5178-42f9-b970-5520f85a060f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:07:28.336827 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43ae6f71-5178-42f9-b970-5520f85a060f admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:07:28.366535 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 934/1846] 10.222.0.50 () {58 vars in 1122 bytes} [Tue Sep 13 16:07:28 2022] GET /identity/v3/roles => generated 1420 bytes in 228 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 16:07:28.371147 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43ae6f71-5178-42f9-b970-5520f85a060f admin admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:07:28.422550 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43ae6f71-5178-42f9-b970-5520f85a060f admin admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:07:28.433524 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 913/1847] 10.222.0.50 () {58 vars in 958 bytes} [Tue Sep 13 16:07:28 2022] POST /identity/v3/auth/tokens => generated 3321 bytes in 280 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:07:28.437768 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72996bd9-4f71-4d17-8905-c6980209bda1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:07:28.438471 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72996bd9-4f71-4d17-8905-c6980209bda1 admin admin] REQUEST_METHOD: `PUT` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:07:28.438755 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72996bd9-4f71-4d17-8905-c6980209bda1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:07:28.441730 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72996bd9-4f71-4d17-8905-c6980209bda1 admin admin] PATH_INFO: `/v3/projects/43c504d755af421f9a78b166e6675495/users/5a0fc30ba5f64e41bc6430a39f4adfdd/roles/031d4e1a23e34f71858fc4a6dfbd4ccb` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:07:28.480946 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72996bd9-4f71-4d17-8905-c6980209bda1 admin admin] RBAC: Authorizing `identity:create_grant(project_id=43c504d755af421f9a78b166e6675495, user_id=5a0fc30ba5f64e41bc6430a39f4adfdd, role_id=031d4e1a23e34f71858fc4a6dfbd4ccb)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:07:28.484870 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72996bd9-4f71-4d17-8905-c6980209bda1 admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:07:28.517200 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 935/1848] 10.222.0.50 () {60 vars in 1483 bytes} [Tue Sep 13 16:07:28 2022] PUT /identity/v3/projects/43c504d755af421f9a78b166e6675495/users/5a0fc30ba5f64e41bc6430a39f4adfdd/roles/031d4e1a23e34f71858fc4a6dfbd4ccb => generated 0 bytes in 143 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 16:07:28.517791 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8491d1da-1666-4498-b492-9ac18631c1cb None None] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:07:28.531969 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79aaa933-3cf9-4f30-8494-88e7beb5e6d9 None None] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:07:28.533154 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79aaa933-3cf9-4f30-8494-88e7beb5e6d9 None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:07:28.533154 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79aaa933-3cf9-4f30-8494-88e7beb5e6d9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:07:28.534988 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8491d1da-1666-4498-b492-9ac18631c1cb None None] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:07:28.580324 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.auth.core [None req-79aaa933-3cf9-4f30-8494-88e7beb5e6d9 None None] MFA Rules not processed for user `4e039b46c5e3445bbb39904b00c5430e`. Rule list: `[]` (Enabled: `True`). {{(pid=81437) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:07:28.584993 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-79aaa933-3cf9-4f30-8494-88e7beb5e6d9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:07:28.652754 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:07:28.695843 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-825998f5-44a3-49dc-928a-a8393929320d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:07:28.696783 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-825998f5-44a3-49dc-928a-a8393929320d admin admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:07:28.697693 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-825998f5-44a3-49dc-928a-a8393929320d admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:07:28.697975 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-825998f5-44a3-49dc-928a-a8393929320d admin admin] PATH_INFO: `/v3/roles` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:07:28.698640 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-825998f5-44a3-49dc-928a-a8393929320d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:07:28.705996 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-825998f5-44a3-49dc-928a-a8393929320d admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:07:28.738611 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 914/1849] 10.222.0.50 () {58 vars in 1122 bytes} [Tue Sep 13 16:07:28 2022] GET /identity/v3/roles => generated 1420 bytes in 221 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 16:07:28.739855 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-825998f5-44a3-49dc-928a-a8393929320d admin admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:07:28.744609 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-825998f5-44a3-49dc-928a-a8393929320d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:07:28.814213 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 936/1850] 10.222.0.50 () {58 vars in 1018 bytes} [Tue Sep 13 16:07:28 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 283 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 13 16:07:28.815331 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79aaa933-3cf9-4f30-8494-88e7beb5e6d9 None None] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:07:28.843057 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79aaa933-3cf9-4f30-8494-88e7beb5e6d9 None None] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:07:28.877030 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29d0e6fe-ce0e-4e79-ab07-7c16ac9fcffc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:07:28.880785 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29d0e6fe-ce0e-4e79-ab07-7c16ac9fcffc admin admin] REQUEST_METHOD: `PUT` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:07:28.882189 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29d0e6fe-ce0e-4e79-ab07-7c16ac9fcffc admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:07:28.882741 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29d0e6fe-ce0e-4e79-ab07-7c16ac9fcffc admin admin] PATH_INFO: `/v3/projects/43c504d755af421f9a78b166e6675495/users/5a0fc30ba5f64e41bc6430a39f4adfdd/roles/031d4e1a23e34f71858fc4a6dfbd4ccb` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:07:28.894460 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29d0e6fe-ce0e-4e79-ab07-7c16ac9fcffc admin admin] RBAC: Authorizing `identity:create_grant(project_id=43c504d755af421f9a78b166e6675495, user_id=5a0fc30ba5f64e41bc6430a39f4adfdd, role_id=031d4e1a23e34f71858fc4a6dfbd4ccb)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:07:28.907096 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29d0e6fe-ce0e-4e79-ab07-7c16ac9fcffc admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:07:28.951185 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 937/1851] 10.222.0.50 () {60 vars in 1483 bytes} [Tue Sep 13 16:07:28 2022] PUT /identity/v3/projects/43c504d755af421f9a78b166e6675495/users/5a0fc30ba5f64e41bc6430a39f4adfdd/roles/031d4e1a23e34f71858fc4a6dfbd4ccb => generated 0 bytes in 136 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 16:07:28.958568 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-825998f5-44a3-49dc-928a-a8393929320d admin admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:07:28.962755 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97cf4fe4-c226-4cb0-ad63-a1649a6b4721 None None] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:07:28.963333 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97cf4fe4-c226-4cb0-ad63-a1649a6b4721 None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:07:28.963738 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97cf4fe4-c226-4cb0-ad63-a1649a6b4721 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:07:29.034367 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.auth.core [None req-97cf4fe4-c226-4cb0-ad63-a1649a6b4721 None None] MFA Rules not processed for user `5a0fc30ba5f64e41bc6430a39f4adfdd`. Rule list: `[]` (Enabled: `True`). {{(pid=81437) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:07:29.038042 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-97cf4fe4-c226-4cb0-ad63-a1649a6b4721 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:07:29.094430 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa1f6ff7-b06a-4af7-abc4-6c88a74020ee service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7fd6b5afd67b48f9b0f0597ccd4cf4df', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:07:29.095464 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa1f6ff7-b06a-4af7-abc4-6c88a74020ee service cinder] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:07:29.096129 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa1f6ff7-b06a-4af7-abc4-6c88a74020ee service cinder] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:07:29.096662 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa1f6ff7-b06a-4af7-abc4-6c88a74020ee service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:07:29.193630 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa1f6ff7-b06a-4af7-abc4-6c88a74020ee service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:07:29.196178 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa1f6ff7-b06a-4af7-abc4-6c88a74020ee service cinder] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:07:29.223346 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 915/1852] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 16:07:28 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 484 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 16:07:29.232277 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa1f6ff7-b06a-4af7-abc4-6c88a74020ee service cinder] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:07:29.233991 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-fa1f6ff7-b06a-4af7-abc4-6c88a74020ee service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:07:29.270459 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:07:29.298140 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 938/1853] 10.222.0.50 () {58 vars in 958 bytes} [Tue Sep 13 16:07:28 2022] POST /identity/v3/auth/tokens => generated 3323 bytes in 337 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 13 16:07:29.322585 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:07:29.354862 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa1f6ff7-b06a-4af7-abc4-6c88a74020ee service cinder] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:07:29.361963 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b78ce6db-d79c-4967-b736-9557070bb11b service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4e039b46c5e3445bbb39904b00c5430e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'admin', 'ResellerAdmin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:07:29.362869 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b78ce6db-d79c-4967-b736-9557070bb11b service nova] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:07:29.363415 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b78ce6db-d79c-4967-b736-9557070bb11b service nova] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:07:29.365294 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b78ce6db-d79c-4967-b736-9557070bb11b service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:07:29.375920 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b78ce6db-d79c-4967-b736-9557070bb11b service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:07:29.378719 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b78ce6db-d79c-4967-b736-9557070bb11b service nova] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:07:29.406931 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97cf4fe4-c226-4cb0-ad63-a1649a6b4721 None None] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:07:29.420016 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97cf4fe4-c226-4cb0-ad63-a1649a6b4721 None None] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:07:29.420606 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 916/1854] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 16:07:29 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 189 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 13 16:07:29.437738 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39c21faf-a4ff-4bf8-bbcd-d81c28cc14fc service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7fd6b5afd67b48f9b0f0597ccd4cf4df', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:07:29.439642 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39c21faf-a4ff-4bf8-bbcd-d81c28cc14fc service cinder] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:07:29.439938 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39c21faf-a4ff-4bf8-bbcd-d81c28cc14fc service cinder] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:07:29.440218 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39c21faf-a4ff-4bf8-bbcd-d81c28cc14fc service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:07:29.481215 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39c21faf-a4ff-4bf8-bbcd-d81c28cc14fc service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:07:29.487611 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39c21faf-a4ff-4bf8-bbcd-d81c28cc14fc service cinder] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:07:29.526835 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 939/1855] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 16:07:29 2022] GET /identity/v3/auth/tokens => generated 3323 bytes in 121 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 16:07:30.553853 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b78ce6db-d79c-4967-b736-9557070bb11b service nova] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:07:30.581440 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b78ce6db-d79c-4967-b736-9557070bb11b service nova] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:07:30.609240 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14c4c99c-3477-46ea-88d0-b77ede4a0fee tempest-VolumeQuotaClassesTest-304922044 tempest-VolumeQuotaClassesTest-304922044-project-admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5a0fc30ba5f64e41bc6430a39f4adfdd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '43c504d755af421f9a78b166e6675495', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:07:30.610212 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14c4c99c-3477-46ea-88d0-b77ede4a0fee tempest-VolumeQuotaClassesTest-304922044 tempest-VolumeQuotaClassesTest-304922044-project-admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:07:30.610643 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14c4c99c-3477-46ea-88d0-b77ede4a0fee tempest-VolumeQuotaClassesTest-304922044 tempest-VolumeQuotaClassesTest-304922044-project-admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:07:30.611089 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14c4c99c-3477-46ea-88d0-b77ede4a0fee tempest-VolumeQuotaClassesTest-304922044 tempest-VolumeQuotaClassesTest-304922044-project-admin] PATH_INFO: `/v3/domains` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:07:30.611903 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14c4c99c-3477-46ea-88d0-b77ede4a0fee tempest-VolumeQuotaClassesTest-304922044 tempest-VolumeQuotaClassesTest-304922044-project-admin] RBAC: Adding query filter params (name=Default) {{(pid=81438) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 16:07:30.612355 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14c4c99c-3477-46ea-88d0-b77ede4a0fee tempest-VolumeQuotaClassesTest-304922044 tempest-VolumeQuotaClassesTest-304922044-project-admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:07:30.614176 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14c4c99c-3477-46ea-88d0-b77ede4a0fee tempest-VolumeQuotaClassesTest-304922044 tempest-VolumeQuotaClassesTest-304922044-project-admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:07:30.661607 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39c21faf-a4ff-4bf8-bbcd-d81c28cc14fc service cinder] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:07:30.662188 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 917/1856] 10.222.0.50 () {58 vars in 1153 bytes} [Tue Sep 13 16:07:30 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 110 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 16:07:30.682457 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39c21faf-a4ff-4bf8-bbcd-d81c28cc14fc service cinder] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:07:30.707412 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c238dfbd-903c-4a92-8b65-ce1e99cf4bdf tempest-VolumeQuotaClassesTest-304922044 tempest-VolumeQuotaClassesTest-304922044-project-admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5a0fc30ba5f64e41bc6430a39f4adfdd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '43c504d755af421f9a78b166e6675495', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:07:30.708188 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c238dfbd-903c-4a92-8b65-ce1e99cf4bdf tempest-VolumeQuotaClassesTest-304922044 tempest-VolumeQuotaClassesTest-304922044-project-admin] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:07:30.708603 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c238dfbd-903c-4a92-8b65-ce1e99cf4bdf tempest-VolumeQuotaClassesTest-304922044 tempest-VolumeQuotaClassesTest-304922044-project-admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:07:30.708968 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c238dfbd-903c-4a92-8b65-ce1e99cf4bdf tempest-VolumeQuotaClassesTest-304922044 tempest-VolumeQuotaClassesTest-304922044-project-admin] PATH_INFO: `/v3/projects` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:07:30.709725 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c238dfbd-903c-4a92-8b65-ce1e99cf4bdf tempest-VolumeQuotaClassesTest-304922044 tempest-VolumeQuotaClassesTest-304922044-project-admin] RBAC: Authorizing `identity:create_project()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:07:30.711808 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c238dfbd-903c-4a92-8b65-ce1e99cf4bdf tempest-VolumeQuotaClassesTest-304922044 tempest-VolumeQuotaClassesTest-304922044-project-admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:07:30.796782 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 940/1857] 10.222.0.50 () {60 vars in 1153 bytes} [Tue Sep 13 16:07:30 2022] POST /identity/v3/projects => generated 354 bytes in 135 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 13 16:07:30.798144 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14c4c99c-3477-46ea-88d0-b77ede4a0fee tempest-VolumeQuotaClassesTest-304922044 tempest-VolumeQuotaClassesTest-304922044-project-admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:07:30.825701 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c238dfbd-903c-4a92-8b65-ce1e99cf4bdf tempest-VolumeQuotaClassesTest-304922044 tempest-VolumeQuotaClassesTest-304922044-project-admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:07:30.831444 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14c4c99c-3477-46ea-88d0-b77ede4a0fee tempest-VolumeQuotaClassesTest-304922044 tempest-VolumeQuotaClassesTest-304922044-project-admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:07:30.976199 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c238dfbd-903c-4a92-8b65-ce1e99cf4bdf tempest-VolumeQuotaClassesTest-304922044 tempest-VolumeQuotaClassesTest-304922044-project-admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:07:30.980732 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d357ef9e-195b-4ad7-b939-27bc3ecd3c1f service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7fd6b5afd67b48f9b0f0597ccd4cf4df', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:07:30.981326 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d357ef9e-195b-4ad7-b939-27bc3ecd3c1f service cinder] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:07:30.981644 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d357ef9e-195b-4ad7-b939-27bc3ecd3c1f service cinder] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:07:30.981884 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d357ef9e-195b-4ad7-b939-27bc3ecd3c1f service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:07:30.991106 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aacc2399-e803-4486-b3fb-2df29b2fa4d9 tempest-VolumeQuotaClassesTest-304922044 tempest-VolumeQuotaClassesTest-304922044-project-admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5a0fc30ba5f64e41bc6430a39f4adfdd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '43c504d755af421f9a78b166e6675495', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:07:30.992331 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aacc2399-e803-4486-b3fb-2df29b2fa4d9 tempest-VolumeQuotaClassesTest-304922044 tempest-VolumeQuotaClassesTest-304922044-project-admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:07:30.992763 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aacc2399-e803-4486-b3fb-2df29b2fa4d9 tempest-VolumeQuotaClassesTest-304922044 tempest-VolumeQuotaClassesTest-304922044-project-admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:07:30.993187 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aacc2399-e803-4486-b3fb-2df29b2fa4d9 tempest-VolumeQuotaClassesTest-304922044 tempest-VolumeQuotaClassesTest-304922044-project-admin] PATH_INFO: `/v3/domains` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:07:30.996022 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aacc2399-e803-4486-b3fb-2df29b2fa4d9 tempest-VolumeQuotaClassesTest-304922044 tempest-VolumeQuotaClassesTest-304922044-project-admin] RBAC: Adding query filter params (name=Default) {{(pid=81438) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 16:07:30.996536 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aacc2399-e803-4486-b3fb-2df29b2fa4d9 tempest-VolumeQuotaClassesTest-304922044 tempest-VolumeQuotaClassesTest-304922044-project-admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:07:30.998368 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aacc2399-e803-4486-b3fb-2df29b2fa4d9 tempest-VolumeQuotaClassesTest-304922044 tempest-VolumeQuotaClassesTest-304922044-project-admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:07:31.015124 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 918/1858] 10.222.0.50 () {58 vars in 1153 bytes} [Tue Sep 13 16:07:30 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 218 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 13 16:07:31.147978 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d357ef9e-195b-4ad7-b939-27bc3ecd3c1f service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:07:31.150644 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d357ef9e-195b-4ad7-b939-27bc3ecd3c1f service cinder] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:07:31.163685 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aacc2399-e803-4486-b3fb-2df29b2fa4d9 tempest-VolumeQuotaClassesTest-304922044 tempest-VolumeQuotaClassesTest-304922044-project-admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:07:31.166247 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 941/1859] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 16:07:30 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 340 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 13 16:07:31.229057 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aacc2399-e803-4486-b3fb-2df29b2fa4d9 tempest-VolumeQuotaClassesTest-304922044 tempest-VolumeQuotaClassesTest-304922044-project-admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:07:31.259129 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c94310b4-0ce8-417b-bd5f-a7b14418f734 tempest-VolumeQuotaClassesTest-304922044 tempest-VolumeQuotaClassesTest-304922044-project-admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5a0fc30ba5f64e41bc6430a39f4adfdd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '43c504d755af421f9a78b166e6675495', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:07:31.260294 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c94310b4-0ce8-417b-bd5f-a7b14418f734 tempest-VolumeQuotaClassesTest-304922044 tempest-VolumeQuotaClassesTest-304922044-project-admin] REQUEST_METHOD: `DELETE` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:07:31.276044 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c94310b4-0ce8-417b-bd5f-a7b14418f734 tempest-VolumeQuotaClassesTest-304922044 tempest-VolumeQuotaClassesTest-304922044-project-admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:07:31.276632 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c94310b4-0ce8-417b-bd5f-a7b14418f734 tempest-VolumeQuotaClassesTest-304922044 tempest-VolumeQuotaClassesTest-304922044-project-admin] PATH_INFO: `/v3/projects/b65d6562e2e042be82cdea4eeb7fd7f2` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:07:31.279817 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c94310b4-0ce8-417b-bd5f-a7b14418f734 tempest-VolumeQuotaClassesTest-304922044 tempest-VolumeQuotaClassesTest-304922044-project-admin] RBAC: Authorizing `identity:delete_project(project_id=b65d6562e2e042be82cdea4eeb7fd7f2)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:07:31.282323 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c94310b4-0ce8-417b-bd5f-a7b14418f734 tempest-VolumeQuotaClassesTest-304922044 tempest-VolumeQuotaClassesTest-304922044-project-admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:07:31.446522 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-c94310b4-0ce8-417b-bd5f-a7b14418f734 tempest-VolumeQuotaClassesTest-304922044 tempest-VolumeQuotaClassesTest-304922044-project-admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'b65d6562e2e042be82cdea4eeb7fd7f2', 'request_id': 'req-c94310b4-0ce8-417b-bd5f-a7b14418f734'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:07:31.463496 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-c94310b4-0ce8-417b-bd5f-a7b14418f734 tempest-VolumeQuotaClassesTest-304922044 tempest-VolumeQuotaClassesTest-304922044-project-admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'b65d6562e2e042be82cdea4eeb7fd7f2', 'request_id': 'req-c94310b4-0ce8-417b-bd5f-a7b14418f734'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:07:31.480750 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-c94310b4-0ce8-417b-bd5f-a7b14418f734 tempest-VolumeQuotaClassesTest-304922044 tempest-VolumeQuotaClassesTest-304922044-project-admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'b65d6562e2e042be82cdea4eeb7fd7f2', 'request_id': 'req-c94310b4-0ce8-417b-bd5f-a7b14418f734'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:07:31.505684 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-c94310b4-0ce8-417b-bd5f-a7b14418f734 tempest-VolumeQuotaClassesTest-304922044 tempest-VolumeQuotaClassesTest-304922044-project-admin] The token cache is being invalidate because project b65d6562e2e042be82cdea4eeb7fd7f2 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81438) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 13 16:07:31.505684 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-c94310b4-0ce8-417b-bd5f-a7b14418f734 tempest-VolumeQuotaClassesTest-304922044 tempest-VolumeQuotaClassesTest-304922044-project-admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:07:31.506526 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-c94310b4-0ce8-417b-bd5f-a7b14418f734 tempest-VolumeQuotaClassesTest-304922044 tempest-VolumeQuotaClassesTest-304922044-project-admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:07:31.513835 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 919/1860] 10.222.0.50 () {58 vars in 1233 bytes} [Tue Sep 13 16:07:31 2022] DELETE /identity/v3/projects/b65d6562e2e042be82cdea4eeb7fd7f2 => generated 0 bytes in 351 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 16:07:32.709666 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d357ef9e-195b-4ad7-b939-27bc3ecd3c1f service cinder] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:07:32.831545 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c94310b4-0ce8-417b-bd5f-a7b14418f734 tempest-VolumeQuotaClassesTest-304922044 tempest-VolumeQuotaClassesTest-304922044-project-admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:07:32.833663 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-c94310b4-0ce8-417b-bd5f-a7b14418f734 tempest-VolumeQuotaClassesTest-304922044 tempest-VolumeQuotaClassesTest-304922044-project-admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:07:32.839669 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d357ef9e-195b-4ad7-b939-27bc3ecd3c1f service cinder] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:07:32.867884 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:07:32.904509 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:07:32.911928 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79ae5813-ccb8-421b-9073-b9f6647b2802 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:07:32.912647 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79ae5813-ccb8-421b-9073-b9f6647b2802 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:07:32.912939 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79ae5813-ccb8-421b-9073-b9f6647b2802 admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:07:32.913207 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79ae5813-ccb8-421b-9073-b9f6647b2802 admin admin] PATH_INFO: `/v3/users/f804092800c54ba0b1addcde7000012c` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:07:32.914374 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79ae5813-ccb8-421b-9073-b9f6647b2802 admin admin] RBAC: Authorizing `identity:delete_user(user_id=f804092800c54ba0b1addcde7000012c)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:07:32.917546 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79ae5813-ccb8-421b-9073-b9f6647b2802 admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:07:32.925006 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c94310b4-0ce8-417b-bd5f-a7b14418f734 tempest-VolumeQuotaClassesTest-304922044 tempest-VolumeQuotaClassesTest-304922044-project-admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:07:32.929498 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-708dade8-49e6-41f4-adef-22be01eb2fde service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7fd6b5afd67b48f9b0f0597ccd4cf4df', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:07:32.930302 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-708dade8-49e6-41f4-adef-22be01eb2fde service cinder] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:07:32.930876 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-708dade8-49e6-41f4-adef-22be01eb2fde service cinder] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:07:32.931267 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-708dade8-49e6-41f4-adef-22be01eb2fde service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:07:32.947405 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-708dade8-49e6-41f4-adef-22be01eb2fde service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:07:32.981602 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-79ae5813-ccb8-421b-9073-b9f6647b2802 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'f804092800c54ba0b1addcde7000012c'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:07:32.989096 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-79ae5813-ccb8-421b-9073-b9f6647b2802 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'f804092800c54ba0b1addcde7000012c'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:07:33.004998 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-79ae5813-ccb8-421b-9073-b9f6647b2802 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'f804092800c54ba0b1addcde7000012c'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:07:33.005752 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-79ae5813-ccb8-421b-9073-b9f6647b2802 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'f804092800c54ba0b1addcde7000012c'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:07:33.014376 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-79ae5813-ccb8-421b-9073-b9f6647b2802 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'f804092800c54ba0b1addcde7000012c'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:07:33.020416 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 942/1861] 10.222.0.50 () {58 vars in 1224 bytes} [Tue Sep 13 16:07:32 2022] DELETE /identity/v3/users/f804092800c54ba0b1addcde7000012c => generated 0 bytes in 312 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 16:07:33.024607 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a46eed71-4823-4b15-b412-40ca1dfa84a3 None None] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:07:33.024974 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a46eed71-4823-4b15-b412-40ca1dfa84a3 None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:07:33.025260 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a46eed71-4823-4b15-b412-40ca1dfa84a3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:07:33.033923 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-708dade8-49e6-41f4-adef-22be01eb2fde service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:07:33.035601 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-708dade8-49e6-41f4-adef-22be01eb2fde service cinder] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:07:33.044544 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.auth.core [None req-a46eed71-4823-4b15-b412-40ca1dfa84a3 None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81437) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:07:33.045826 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-a46eed71-4823-4b15-b412-40ca1dfa84a3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:07:33.050307 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 920/1862] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 16:07:32 2022] GET /identity/v3/auth/tokens => generated 3321 bytes in 219 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 13 16:07:33.110802 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 943/1863] 10.222.0.50 () {58 vars in 958 bytes} [Tue Sep 13 16:07:33 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:07:33.123693 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-708dade8-49e6-41f4-adef-22be01eb2fde service cinder] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:07:33.125762 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-708dade8-49e6-41f4-adef-22be01eb2fde service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:07:33.193052 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-708dade8-49e6-41f4-adef-22be01eb2fde service cinder] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:07:33.250225 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-155dd849-f2b5-42bb-8273-acc365a1e938 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f436f07939d4855a21b195bc7566244', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:07:33.250841 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-155dd849-f2b5-42bb-8273-acc365a1e938 service neutron] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:07:33.251266 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-155dd849-f2b5-42bb-8273-acc365a1e938 service neutron] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:07:33.252218 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-155dd849-f2b5-42bb-8273-acc365a1e938 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:07:33.265336 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-155dd849-f2b5-42bb-8273-acc365a1e938 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:07:33.266930 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-155dd849-f2b5-42bb-8273-acc365a1e938 service neutron] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:07:33.277782 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 921/1864] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 16:07:33 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 155 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 16:07:33.648617 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a46eed71-4823-4b15-b412-40ca1dfa84a3 None None] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:07:33.650475 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-a46eed71-4823-4b15-b412-40ca1dfa84a3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:07:33.668678 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a46eed71-4823-4b15-b412-40ca1dfa84a3 None None] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:07:33.677047 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed7be701-051a-4519-aa76-a757170f318e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:07:33.677856 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed7be701-051a-4519-aa76-a757170f318e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:07:33.678236 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed7be701-051a-4519-aa76-a757170f318e admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:07:33.678650 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed7be701-051a-4519-aa76-a757170f318e admin admin] PATH_INFO: `/v3/projects/d0c3c872b5d74321b4e09f8d170a2207` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:07:33.680415 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed7be701-051a-4519-aa76-a757170f318e admin admin] RBAC: Authorizing `identity:delete_project(project_id=d0c3c872b5d74321b4e09f8d170a2207)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:07:33.683406 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed7be701-051a-4519-aa76-a757170f318e admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:07:33.765081 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=86,keystone:DELETE=13,keystone:INSERT=1 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:07:33.772635 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-ed7be701-051a-4519-aa76-a757170f318e admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'd0c3c872b5d74321b4e09f8d170a2207', 'request_id': 'req-ed7be701-051a-4519-aa76-a757170f318e'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:07:33.798555 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-ed7be701-051a-4519-aa76-a757170f318e admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'd0c3c872b5d74321b4e09f8d170a2207', 'request_id': 'req-ed7be701-051a-4519-aa76-a757170f318e'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:07:33.807917 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-ed7be701-051a-4519-aa76-a757170f318e admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'd0c3c872b5d74321b4e09f8d170a2207', 'request_id': 'req-ed7be701-051a-4519-aa76-a757170f318e'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:07:33.828331 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-ed7be701-051a-4519-aa76-a757170f318e admin admin] The token cache is being invalidate because project d0c3c872b5d74321b4e09f8d170a2207 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81437) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 13 16:07:33.829034 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-ed7be701-051a-4519-aa76-a757170f318e admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:07:33.829864 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-ed7be701-051a-4519-aa76-a757170f318e admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:07:33.831596 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 944/1865] 10.222.0.50 () {58 vars in 1233 bytes} [Tue Sep 13 16:07:33 2022] DELETE /identity/v3/projects/d0c3c872b5d74321b4e09f8d170a2207 => generated 0 bytes in 183 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 16:07:33.835383 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-155dd849-f2b5-42bb-8273-acc365a1e938 service neutron] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:07:33.846586 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-155dd849-f2b5-42bb-8273-acc365a1e938 service neutron] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:07:33.895838 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:07:33.910327 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18e6c724-8200-4c6f-8284-451460c3b1d7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:07:33.911003 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18e6c724-8200-4c6f-8284-451460c3b1d7 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:07:33.911725 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18e6c724-8200-4c6f-8284-451460c3b1d7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:07:33.912004 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18e6c724-8200-4c6f-8284-451460c3b1d7 admin admin] PATH_INFO: `/v3/users/5a0fc30ba5f64e41bc6430a39f4adfdd` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:07:33.913190 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18e6c724-8200-4c6f-8284-451460c3b1d7 admin admin] RBAC: Authorizing `identity:delete_user(user_id=5a0fc30ba5f64e41bc6430a39f4adfdd)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:07:33.915728 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18e6c724-8200-4c6f-8284-451460c3b1d7 admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:07:33.963405 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-18e6c724-8200-4c6f-8284-451460c3b1d7 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '5a0fc30ba5f64e41bc6430a39f4adfdd'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:07:33.971899 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-18e6c724-8200-4c6f-8284-451460c3b1d7 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '5a0fc30ba5f64e41bc6430a39f4adfdd'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:07:33.984402 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-18e6c724-8200-4c6f-8284-451460c3b1d7 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '5a0fc30ba5f64e41bc6430a39f4adfdd'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:07:33.985038 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-18e6c724-8200-4c6f-8284-451460c3b1d7 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '5a0fc30ba5f64e41bc6430a39f4adfdd'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:07:33.991829 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-18e6c724-8200-4c6f-8284-451460c3b1d7 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '5a0fc30ba5f64e41bc6430a39f4adfdd'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:07:33.998111 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 922/1866] 10.222.0.50 () {58 vars in 1224 bytes} [Tue Sep 13 16:07:33 2022] DELETE /identity/v3/users/5a0fc30ba5f64e41bc6430a39f4adfdd => generated 0 bytes in 163 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 16:07:34.270967 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed7be701-051a-4519-aa76-a757170f318e admin admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:07:34.272498 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-ed7be701-051a-4519-aa76-a757170f318e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:07:34.287594 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed7be701-051a-4519-aa76-a757170f318e admin admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:07:34.355229 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de0c3473-d4ee-4173-ba48-447be302ae59 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:07:34.355879 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de0c3473-d4ee-4173-ba48-447be302ae59 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:07:34.356161 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de0c3473-d4ee-4173-ba48-447be302ae59 admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:07:34.356433 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de0c3473-d4ee-4173-ba48-447be302ae59 admin admin] PATH_INFO: `/v3/projects/43c504d755af421f9a78b166e6675495` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:07:34.357591 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de0c3473-d4ee-4173-ba48-447be302ae59 admin admin] RBAC: Authorizing `identity:delete_project(project_id=43c504d755af421f9a78b166e6675495)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:07:34.359575 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de0c3473-d4ee-4173-ba48-447be302ae59 admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:07:34.408884 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-de0c3473-d4ee-4173-ba48-447be302ae59 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '43c504d755af421f9a78b166e6675495', 'request_id': 'req-de0c3473-d4ee-4173-ba48-447be302ae59'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:07:34.421269 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-de0c3473-d4ee-4173-ba48-447be302ae59 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '43c504d755af421f9a78b166e6675495', 'request_id': 'req-de0c3473-d4ee-4173-ba48-447be302ae59'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:07:34.428828 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-de0c3473-d4ee-4173-ba48-447be302ae59 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '43c504d755af421f9a78b166e6675495', 'request_id': 'req-de0c3473-d4ee-4173-ba48-447be302ae59'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:07:34.440071 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-de0c3473-d4ee-4173-ba48-447be302ae59 admin admin] The token cache is being invalidate because project 43c504d755af421f9a78b166e6675495 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81437) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 13 16:07:34.440360 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-de0c3473-d4ee-4173-ba48-447be302ae59 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:07:34.441010 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-de0c3473-d4ee-4173-ba48-447be302ae59 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:07:34.442854 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 945/1867] 10.222.0.50 () {58 vars in 1233 bytes} [Tue Sep 13 16:07:34 2022] DELETE /identity/v3/projects/43c504d755af421f9a78b166e6675495 => generated 0 bytes in 172 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 16:07:34.447774 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-982a0eb9-f9fd-4026-922c-de1b396fba0d None None] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:07:34.448159 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-982a0eb9-f9fd-4026-922c-de1b396fba0d None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:07:34.448451 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-982a0eb9-f9fd-4026-922c-de1b396fba0d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:07:34.469913 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.auth.core [None req-982a0eb9-f9fd-4026-922c-de1b396fba0d None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81438) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:07:34.471249 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-982a0eb9-f9fd-4026-922c-de1b396fba0d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:07:34.534944 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 923/1868] 10.222.0.50 () {58 vars in 958 bytes} [Tue Sep 13 16:07:34 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:07:34.543456 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31a52173-bf43-4c02-a465-c508185a4de8 None None] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:07:34.543742 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31a52173-bf43-4c02-a465-c508185a4de8 None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:07:34.544013 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31a52173-bf43-4c02-a465-c508185a4de8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:07:34.568872 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.auth.core [None req-31a52173-bf43-4c02-a465-c508185a4de8 None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81437) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:07:34.570073 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-31a52173-bf43-4c02-a465-c508185a4de8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:07:34.578621 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 946/1869] 10.222.0.50 () {58 vars in 958 bytes} [Tue Sep 13 16:07:34 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 36 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:07:34.582930 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-982a0eb9-f9fd-4026-922c-de1b396fba0d None None] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:07:34.593606 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-982a0eb9-f9fd-4026-922c-de1b396fba0d None None] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:07:34.598180 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03c5b05a-3c37-411d-97a2-c8ee0614d781 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:07:34.598779 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03c5b05a-3c37-411d-97a2-c8ee0614d781 admin admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:07:34.599088 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03c5b05a-3c37-411d-97a2-c8ee0614d781 admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:07:34.599356 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03c5b05a-3c37-411d-97a2-c8ee0614d781 admin admin] PATH_INFO: `/v3/domains` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:07:34.599991 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03c5b05a-3c37-411d-97a2-c8ee0614d781 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81438) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 16:07:34.600293 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03c5b05a-3c37-411d-97a2-c8ee0614d781 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:07:34.601906 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03c5b05a-3c37-411d-97a2-c8ee0614d781 admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:07:34.617676 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 924/1870] 10.222.0.50 () {58 vars in 1153 bytes} [Tue Sep 13 16:07:34 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 16:07:34.619406 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31a52173-bf43-4c02-a465-c508185a4de8 None None] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:07:34.628646 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31a52173-bf43-4c02-a465-c508185a4de8 None None] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:07:34.633277 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4051bd74-4d2e-423a-abc1-ac85ac4112e5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:07:34.633904 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4051bd74-4d2e-423a-abc1-ac85ac4112e5 admin admin] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:07:34.634969 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4051bd74-4d2e-423a-abc1-ac85ac4112e5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:07:34.635403 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4051bd74-4d2e-423a-abc1-ac85ac4112e5 admin admin] PATH_INFO: `/v3/projects` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:07:34.636038 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4051bd74-4d2e-423a-abc1-ac85ac4112e5 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:07:34.637910 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4051bd74-4d2e-423a-abc1-ac85ac4112e5 admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:07:34.664035 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 947/1871] 10.222.0.50 () {60 vars in 1153 bytes} [Tue Sep 13 16:07:34 2022] POST /identity/v3/projects => generated 409 bytes in 46 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 13 16:07:34.668861 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03c5b05a-3c37-411d-97a2-c8ee0614d781 admin admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:07:34.678626 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03c5b05a-3c37-411d-97a2-c8ee0614d781 admin admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:07:34.728393 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:07:34.735943 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4de512ae-4ea7-49f0-b00d-33e9131d0df5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:07:34.736546 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4de512ae-4ea7-49f0-b00d-33e9131d0df5 admin admin] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:07:34.736832 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4de512ae-4ea7-49f0-b00d-33e9131d0df5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:07:34.737143 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4de512ae-4ea7-49f0-b00d-33e9131d0df5 admin admin] PATH_INFO: `/v3/users` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:07:34.738212 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4de512ae-4ea7-49f0-b00d-33e9131d0df5 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:07:34.740350 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4de512ae-4ea7-49f0-b00d-33e9131d0df5 admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:07:34.765196 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 925/1872] 10.222.0.50 () {60 vars in 1144 bytes} [Tue Sep 13 16:07:34 2022] POST /identity/v3/users => generated 305 bytes in 94 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 16:07:34.766516 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4051bd74-4d2e-423a-abc1-ac85ac4112e5 admin admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:07:34.777384 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4051bd74-4d2e-423a-abc1-ac85ac4112e5 admin admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:07:34.782349 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-600a2716-9f56-48d4-bbd2-23aa902aa21e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:07:34.782987 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-600a2716-9f56-48d4-bbd2-23aa902aa21e admin admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:07:34.783271 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-600a2716-9f56-48d4-bbd2-23aa902aa21e admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:07:34.783550 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-600a2716-9f56-48d4-bbd2-23aa902aa21e admin admin] PATH_INFO: `/v3/roles` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:07:34.784194 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-600a2716-9f56-48d4-bbd2-23aa902aa21e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:07:34.785797 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-600a2716-9f56-48d4-bbd2-23aa902aa21e admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:07:34.799753 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 948/1873] 10.222.0.50 () {58 vars in 1122 bytes} [Tue Sep 13 16:07:34 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 16:07:34.801676 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4de512ae-4ea7-49f0-b00d-33e9131d0df5 admin admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:07:34.812753 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4de512ae-4ea7-49f0-b00d-33e9131d0df5 admin admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:07:34.817499 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7eb0c14f-a01a-4bff-882e-5318bc2fa6de admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:07:34.818114 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7eb0c14f-a01a-4bff-882e-5318bc2fa6de admin admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:07:34.818394 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7eb0c14f-a01a-4bff-882e-5318bc2fa6de admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:07:34.818670 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7eb0c14f-a01a-4bff-882e-5318bc2fa6de admin admin] PATH_INFO: `/v3/roles` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:07:34.819366 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7eb0c14f-a01a-4bff-882e-5318bc2fa6de admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:07:34.821080 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7eb0c14f-a01a-4bff-882e-5318bc2fa6de admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:07:34.832078 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 926/1874] 10.222.0.50 () {58 vars in 1122 bytes} [Tue Sep 13 16:07:34 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 16:07:34.836588 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-600a2716-9f56-48d4-bbd2-23aa902aa21e admin admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:07:34.846660 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-600a2716-9f56-48d4-bbd2-23aa902aa21e admin admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:07:34.851431 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4890b91-62b8-42fc-8e45-fc858a2a0bd5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:07:34.852114 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4890b91-62b8-42fc-8e45-fc858a2a0bd5 admin admin] REQUEST_METHOD: `PUT` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:07:34.852348 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4890b91-62b8-42fc-8e45-fc858a2a0bd5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:07:34.853162 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4890b91-62b8-42fc-8e45-fc858a2a0bd5 admin admin] PATH_INFO: `/v3/projects/8c14445158b3411bb08c888abafa14cd/users/1acf5f9088dd442b8c18e290f175fb64/roles/15db4aaca3264ae29bde38903ba43471` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:07:34.868330 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4890b91-62b8-42fc-8e45-fc858a2a0bd5 admin admin] RBAC: Authorizing `identity:create_grant(project_id=8c14445158b3411bb08c888abafa14cd, user_id=1acf5f9088dd442b8c18e290f175fb64, role_id=15db4aaca3264ae29bde38903ba43471)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:07:34.871947 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4890b91-62b8-42fc-8e45-fc858a2a0bd5 admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:07:34.875405 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=85,keystone:DELETE=11,keystone:INSERT=4 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:07:34.895110 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 949/1875] 10.222.0.50 () {60 vars in 1483 bytes} [Tue Sep 13 16:07:34 2022] PUT /identity/v3/projects/8c14445158b3411bb08c888abafa14cd/users/1acf5f9088dd442b8c18e290f175fb64/roles/15db4aaca3264ae29bde38903ba43471 => generated 0 bytes in 53 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 16:07:34.899994 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4049130e-bf16-4836-951a-e796cde21206 None None] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:07:34.900590 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4049130e-bf16-4836-951a-e796cde21206 None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:07:34.900590 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4049130e-bf16-4836-951a-e796cde21206 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:07:34.930870 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.auth.core [None req-4049130e-bf16-4836-951a-e796cde21206 None None] MFA Rules not processed for user `1acf5f9088dd442b8c18e290f175fb64`. Rule list: `[]` (Enabled: `True`). {{(pid=81438) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:07:34.932638 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-4049130e-bf16-4836-951a-e796cde21206 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:07:34.998954 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 927/1876] 10.222.0.50 () {58 vars in 958 bytes} [Tue Sep 13 16:07:34 2022] POST /identity/v3/auth/tokens => generated 3286 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 13 16:07:35.003555 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4890b91-62b8-42fc-8e45-fc858a2a0bd5 admin admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:07:35.083624 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4890b91-62b8-42fc-8e45-fc858a2a0bd5 admin admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:07:35.088545 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fc29996-f73c-4a66-af30-accc012a95d1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:07:35.089161 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fc29996-f73c-4a66-af30-accc012a95d1 admin admin] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:07:35.089471 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fc29996-f73c-4a66-af30-accc012a95d1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:07:35.089713 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fc29996-f73c-4a66-af30-accc012a95d1 admin admin] PATH_INFO: `/v3/projects` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:07:35.090346 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fc29996-f73c-4a66-af30-accc012a95d1 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:07:35.092305 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fc29996-f73c-4a66-af30-accc012a95d1 admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:07:35.141277 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 950/1877] 10.222.0.50 () {60 vars in 1153 bytes} [Tue Sep 13 16:07:35 2022] POST /identity/v3/projects => generated 407 bytes in 137 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 16:07:35.144416 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4049130e-bf16-4836-951a-e796cde21206 None None] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:07:35.155694 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4049130e-bf16-4836-951a-e796cde21206 None None] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:07:35.218653 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c06b22f6-e785-4413-b539-03b8b7f20bb1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:07:35.219306 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c06b22f6-e785-4413-b539-03b8b7f20bb1 admin admin] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:07:35.219588 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c06b22f6-e785-4413-b539-03b8b7f20bb1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:07:35.219860 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c06b22f6-e785-4413-b539-03b8b7f20bb1 admin admin] PATH_INFO: `/v3/users` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:07:35.220488 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c06b22f6-e785-4413-b539-03b8b7f20bb1 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:07:35.222431 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c06b22f6-e785-4413-b539-03b8b7f20bb1 admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:07:35.245512 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 928/1878] 10.222.0.50 () {60 vars in 1144 bytes} [Tue Sep 13 16:07:35 2022] POST /identity/v3/users => generated 310 bytes in 101 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 16:07:35.249461 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fc29996-f73c-4a66-af30-accc012a95d1 admin admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:07:35.260343 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fc29996-f73c-4a66-af30-accc012a95d1 admin admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:07:35.265302 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-def56173-5662-4cb6-8f88-7035c3e65715 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:07:35.265835 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-def56173-5662-4cb6-8f88-7035c3e65715 admin admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:07:35.266116 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-def56173-5662-4cb6-8f88-7035c3e65715 admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:07:35.266389 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-def56173-5662-4cb6-8f88-7035c3e65715 admin admin] PATH_INFO: `/v3/roles` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:07:35.267064 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-def56173-5662-4cb6-8f88-7035c3e65715 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:07:35.268681 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-def56173-5662-4cb6-8f88-7035c3e65715 admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:07:35.280357 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 951/1879] 10.222.0.50 () {58 vars in 1122 bytes} [Tue Sep 13 16:07:35 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 16:07:35.284530 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c06b22f6-e785-4413-b539-03b8b7f20bb1 admin admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:07:35.295306 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c06b22f6-e785-4413-b539-03b8b7f20bb1 admin admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:07:35.299955 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9477dc84-272f-43d9-aa3d-ceb7f9bb92a3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:07:35.300616 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9477dc84-272f-43d9-aa3d-ceb7f9bb92a3 admin admin] REQUEST_METHOD: `PUT` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:07:35.300900 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9477dc84-272f-43d9-aa3d-ceb7f9bb92a3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:07:35.301179 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9477dc84-272f-43d9-aa3d-ceb7f9bb92a3 admin admin] PATH_INFO: `/v3/projects/a9165453186c46ae9d9a80b7aace88b2/users/c29e087c21ba49d5ae3e5031fd500fda/roles/031d4e1a23e34f71858fc4a6dfbd4ccb` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:07:35.316508 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9477dc84-272f-43d9-aa3d-ceb7f9bb92a3 admin admin] RBAC: Authorizing `identity:create_grant(project_id=a9165453186c46ae9d9a80b7aace88b2, user_id=c29e087c21ba49d5ae3e5031fd500fda, role_id=031d4e1a23e34f71858fc4a6dfbd4ccb)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:07:35.319175 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9477dc84-272f-43d9-aa3d-ceb7f9bb92a3 admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:07:35.331902 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 929/1880] 10.222.0.50 () {60 vars in 1483 bytes} [Tue Sep 13 16:07:35 2022] PUT /identity/v3/projects/a9165453186c46ae9d9a80b7aace88b2/users/c29e087c21ba49d5ae3e5031fd500fda/roles/031d4e1a23e34f71858fc4a6dfbd4ccb => generated 0 bytes in 48 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 16:07:35.336018 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-def56173-5662-4cb6-8f88-7035c3e65715 admin admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:07:35.346684 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-def56173-5662-4cb6-8f88-7035c3e65715 admin admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:07:35.402570 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-953829c0-1a7d-4dcd-95b2-c58864d2271e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:07:35.403203 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-953829c0-1a7d-4dcd-95b2-c58864d2271e admin admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:07:35.403483 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-953829c0-1a7d-4dcd-95b2-c58864d2271e admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:07:35.403754 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-953829c0-1a7d-4dcd-95b2-c58864d2271e admin admin] PATH_INFO: `/v3/roles` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:07:35.404384 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-953829c0-1a7d-4dcd-95b2-c58864d2271e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:07:35.405976 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-953829c0-1a7d-4dcd-95b2-c58864d2271e admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:07:35.415762 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 952/1881] 10.222.0.50 () {58 vars in 1122 bytes} [Tue Sep 13 16:07:35 2022] GET /identity/v3/roles => generated 1420 bytes in 80 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 16:07:35.419975 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9477dc84-272f-43d9-aa3d-ceb7f9bb92a3 admin admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:07:35.429956 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9477dc84-272f-43d9-aa3d-ceb7f9bb92a3 admin admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:07:35.434517 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-128fc3e6-ab48-4ff0-92e0-e0e95f75e946 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:07:35.435188 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-128fc3e6-ab48-4ff0-92e0-e0e95f75e946 admin admin] REQUEST_METHOD: `PUT` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:07:35.435464 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-128fc3e6-ab48-4ff0-92e0-e0e95f75e946 admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:07:35.435730 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-128fc3e6-ab48-4ff0-92e0-e0e95f75e946 admin admin] PATH_INFO: `/v3/projects/a9165453186c46ae9d9a80b7aace88b2/users/c29e087c21ba49d5ae3e5031fd500fda/roles/031d4e1a23e34f71858fc4a6dfbd4ccb` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:07:35.437578 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-128fc3e6-ab48-4ff0-92e0-e0e95f75e946 admin admin] RBAC: Authorizing `identity:create_grant(project_id=a9165453186c46ae9d9a80b7aace88b2, user_id=c29e087c21ba49d5ae3e5031fd500fda, role_id=031d4e1a23e34f71858fc4a6dfbd4ccb)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:07:35.440215 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-128fc3e6-ab48-4ff0-92e0-e0e95f75e946 admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:07:35.451140 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 930/1882] 10.222.0.50 () {60 vars in 1483 bytes} [Tue Sep 13 16:07:35 2022] PUT /identity/v3/projects/a9165453186c46ae9d9a80b7aace88b2/users/c29e087c21ba49d5ae3e5031fd500fda/roles/031d4e1a23e34f71858fc4a6dfbd4ccb => generated 0 bytes in 32 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 13 16:07:35.462709 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc47b047-b62d-45c0-a426-821035c27f32 None None] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:07:35.463173 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc47b047-b62d-45c0-a426-821035c27f32 None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:07:35.463401 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc47b047-b62d-45c0-a426-821035c27f32 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:07:35.496414 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.auth.core [None req-cc47b047-b62d-45c0-a426-821035c27f32 None None] MFA Rules not processed for user `c29e087c21ba49d5ae3e5031fd500fda`. Rule list: `[]` (Enabled: `True`). {{(pid=81437) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:07:35.497617 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-cc47b047-b62d-45c0-a426-821035c27f32 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:07:35.564194 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:07:35.595817 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 953/1883] 10.222.0.50 () {58 vars in 958 bytes} [Tue Sep 13 16:07:35 2022] POST /identity/v3/auth/tokens => generated 3351 bytes in 135 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:07:35.603891 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-128fc3e6-ab48-4ff0-92e0-e0e95f75e946 admin admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:07:35.605757 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-128fc3e6-ab48-4ff0-92e0-e0e95f75e946 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:07:35.625743 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=92,keystone:INSERT=8 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:07:35.673630 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-128fc3e6-ab48-4ff0-92e0-e0e95f75e946 admin admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:07:35.677449 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a53ec8b2-3bf1-4c32-af73-d71f49d610a1 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7fd6b5afd67b48f9b0f0597ccd4cf4df', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:07:35.678046 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a53ec8b2-3bf1-4c32-af73-d71f49d610a1 service cinder] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:07:35.678316 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a53ec8b2-3bf1-4c32-af73-d71f49d610a1 service cinder] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:07:35.678582 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a53ec8b2-3bf1-4c32-af73-d71f49d610a1 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:07:35.688508 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a53ec8b2-3bf1-4c32-af73-d71f49d610a1 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:07:35.689942 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a53ec8b2-3bf1-4c32-af73-d71f49d610a1 service cinder] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:07:35.700389 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 931/1884] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 16:07:35 2022] GET /identity/v3/auth/tokens => generated 3351 bytes in 98 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 16:07:35.861750 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc47b047-b62d-45c0-a426-821035c27f32 None None] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:07:35.871509 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc47b047-b62d-45c0-a426-821035c27f32 None None] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:07:35.875171 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-550a62f1-d7ae-4fe6-9167-cd6f1d6755f7 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7fd6b5afd67b48f9b0f0597ccd4cf4df', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:07:35.875774 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-550a62f1-d7ae-4fe6-9167-cd6f1d6755f7 service cinder] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:07:35.876051 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-550a62f1-d7ae-4fe6-9167-cd6f1d6755f7 service cinder] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:07:35.876365 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-550a62f1-d7ae-4fe6-9167-cd6f1d6755f7 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:07:35.878044 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-550a62f1-d7ae-4fe6-9167-cd6f1d6755f7 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:07:35.897570 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a53ec8b2-3bf1-4c32-af73-d71f49d610a1 service cinder] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:07:35.898966 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-a53ec8b2-3bf1-4c32-af73-d71f49d610a1 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:07:35.910361 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a53ec8b2-3bf1-4c32-af73-d71f49d610a1 service cinder] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:07:35.913972 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4aa883df-fca0-468e-ba2e-0cc76b3c683b service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7fd6b5afd67b48f9b0f0597ccd4cf4df', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:07:35.914559 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4aa883df-fca0-468e-ba2e-0cc76b3c683b service cinder] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:07:35.914838 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4aa883df-fca0-468e-ba2e-0cc76b3c683b service cinder] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:07:35.916214 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4aa883df-fca0-468e-ba2e-0cc76b3c683b service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:07:35.945122 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-550a62f1-d7ae-4fe6-9167-cd6f1d6755f7 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:07:35.947383 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-550a62f1-d7ae-4fe6-9167-cd6f1d6755f7 service cinder] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:07:35.957389 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 954/1885] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 16:07:35 2022] GET /identity/v3/auth/tokens => generated 3321 bytes in 96 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 16:07:35.977087 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4aa883df-fca0-468e-ba2e-0cc76b3c683b service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:07:35.978770 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4aa883df-fca0-468e-ba2e-0cc76b3c683b service cinder] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:07:35.987977 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 932/1886] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 16:07:35 2022] GET /identity/v3/auth/tokens => generated 3286 bytes in 91 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 13 16:07:45.953014 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=34 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:07:45.983667 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=47 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:07:49.157424 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-935fc056-abaa-4203-b7b3-f3f671a513b1 None None] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:07:49.158496 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-935fc056-abaa-4203-b7b3-f3f671a513b1 None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:07:49.158676 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-935fc056-abaa-4203-b7b3-f3f671a513b1 None None] PATH_INFO: `/` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:07:49.160489 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 955/1887] 10.222.0.50 () {54 vars in 921 bytes} [Tue Sep 13 16:07:49 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 16:07:49.170002 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6f35e49-94ef-4513-adbe-d2b83ce5a74b None None] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:07:49.170488 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6f35e49-94ef-4513-adbe-d2b83ce5a74b None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:07:49.170581 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6f35e49-94ef-4513-adbe-d2b83ce5a74b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:07:49.191951 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.auth.core [None req-e6f35e49-94ef-4513-adbe-d2b83ce5a74b None None] MFA Rules not processed for user `4e039b46c5e3445bbb39904b00c5430e`. Rule list: `[]` (Enabled: `True`). {{(pid=81438) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:07:49.193183 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-e6f35e49-94ef-4513-adbe-d2b83ce5a74b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:07:49.281023 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 933/1888] 10.222.0.50 () {58 vars in 1018 bytes} [Tue Sep 13 16:07:49 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 111 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:07:49.455568 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-935fc056-abaa-4203-b7b3-f3f671a513b1 None None] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:07:49.457334 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-935fc056-abaa-4203-b7b3-f3f671a513b1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:07:49.475949 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-935fc056-abaa-4203-b7b3-f3f671a513b1 None None] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:07:49.481972 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6a508bb-631b-4569-9954-fabf0340fe29 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4e039b46c5e3445bbb39904b00c5430e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'admin', 'ResellerAdmin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:07:49.482576 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6a508bb-631b-4569-9954-fabf0340fe29 service nova] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:07:49.482860 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6a508bb-631b-4569-9954-fabf0340fe29 service nova] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:07:49.483240 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6a508bb-631b-4569-9954-fabf0340fe29 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:07:49.493362 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6a508bb-631b-4569-9954-fabf0340fe29 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:07:49.494987 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6a508bb-631b-4569-9954-fabf0340fe29 service nova] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:07:49.505956 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 956/1889] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 16:07:49 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 51 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 16:07:53.380624 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6f35e49-94ef-4513-adbe-d2b83ce5a74b None None] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:07:53.383679 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-e6f35e49-94ef-4513-adbe-d2b83ce5a74b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:07:53.450219 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6a508bb-631b-4569-9954-fabf0340fe29 service nova] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:07:53.464405 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6f35e49-94ef-4513-adbe-d2b83ce5a74b None None] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:07:53.468950 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f099b0c8-b956-4737-8f70-2ba6a8d90689 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:07:53.469585 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f099b0c8-b956-4737-8f70-2ba6a8d90689 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:07:53.469866 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f099b0c8-b956-4737-8f70-2ba6a8d90689 admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:07:53.470142 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f099b0c8-b956-4737-8f70-2ba6a8d90689 admin admin] PATH_INFO: `/v3/users/f43c42e067a74df3ba8f477da7780f1e` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:07:53.471374 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f099b0c8-b956-4737-8f70-2ba6a8d90689 admin admin] RBAC: Authorizing `identity:delete_user(user_id=f43c42e067a74df3ba8f477da7780f1e)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:07:53.473280 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f099b0c8-b956-4737-8f70-2ba6a8d90689 admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:07:53.515179 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6a508bb-631b-4569-9954-fabf0340fe29 service nova] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:07:53.519700 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84e08995-290d-4178-ae86-990c19178a77 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:07:53.520322 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84e08995-290d-4178-ae86-990c19178a77 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:07:53.520589 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84e08995-290d-4178-ae86-990c19178a77 admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:07:53.520867 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84e08995-290d-4178-ae86-990c19178a77 admin admin] PATH_INFO: `/v3/users/1acf5f9088dd442b8c18e290f175fb64` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:07:53.521969 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84e08995-290d-4178-ae86-990c19178a77 admin admin] RBAC: Authorizing `identity:delete_user(user_id=1acf5f9088dd442b8c18e290f175fb64)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:07:53.522434 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-f099b0c8-b956-4737-8f70-2ba6a8d90689 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'f43c42e067a74df3ba8f477da7780f1e'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:07:53.523946 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84e08995-290d-4178-ae86-990c19178a77 admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:07:53.531040 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-f099b0c8-b956-4737-8f70-2ba6a8d90689 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'f43c42e067a74df3ba8f477da7780f1e'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:07:53.544328 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-f099b0c8-b956-4737-8f70-2ba6a8d90689 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'f43c42e067a74df3ba8f477da7780f1e'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:07:53.545000 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-f099b0c8-b956-4737-8f70-2ba6a8d90689 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'f43c42e067a74df3ba8f477da7780f1e'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:07:53.552873 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-f099b0c8-b956-4737-8f70-2ba6a8d90689 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'f43c42e067a74df3ba8f477da7780f1e'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:07:53.560575 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 934/1890] 10.222.0.50 () {58 vars in 1224 bytes} [Tue Sep 13 16:07:53 2022] DELETE /identity/v3/users/f43c42e067a74df3ba8f477da7780f1e => generated 0 bytes in 180 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 16:07:53.564887 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0d233e7-2e50-4c4c-992a-357ac525965f None None] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:07:53.565180 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0d233e7-2e50-4c4c-992a-357ac525965f None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:07:53.565457 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0d233e7-2e50-4c4c-992a-357ac525965f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:07:53.575980 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-84e08995-290d-4178-ae86-990c19178a77 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '1acf5f9088dd442b8c18e290f175fb64'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:07:53.583564 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-84e08995-290d-4178-ae86-990c19178a77 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '1acf5f9088dd442b8c18e290f175fb64'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:07:53.585622 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.auth.core [None req-a0d233e7-2e50-4c4c-992a-357ac525965f None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81438) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:07:53.586961 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-a0d233e7-2e50-4c4c-992a-357ac525965f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:07:53.592402 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:07:53.596173 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-84e08995-290d-4178-ae86-990c19178a77 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '1acf5f9088dd442b8c18e290f175fb64'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:07:53.596861 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-84e08995-290d-4178-ae86-990c19178a77 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '1acf5f9088dd442b8c18e290f175fb64'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:07:53.605051 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-84e08995-290d-4178-ae86-990c19178a77 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '1acf5f9088dd442b8c18e290f175fb64'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:07:53.611860 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 957/1891] 10.222.0.50 () {58 vars in 1224 bytes} [Tue Sep 13 16:07:53 2022] DELETE /identity/v3/users/1acf5f9088dd442b8c18e290f175fb64 => generated 0 bytes in 162 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 16:07:53.615822 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8791a1a2-2e78-4c21-9ba5-828f806dcc8a None None] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:07:53.616113 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8791a1a2-2e78-4c21-9ba5-828f806dcc8a None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:07:53.616393 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8791a1a2-2e78-4c21-9ba5-828f806dcc8a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:07:53.637531 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.auth.core [None req-8791a1a2-2e78-4c21-9ba5-828f806dcc8a None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81437) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:07:53.638715 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-8791a1a2-2e78-4c21-9ba5-828f806dcc8a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:07:53.651912 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 935/1892] 10.222.0.50 () {58 vars in 958 bytes} [Tue Sep 13 16:07:53 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 13 16:07:53.662027 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0d233e7-2e50-4c4c-992a-357ac525965f None None] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:07:53.664435 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-a0d233e7-2e50-4c4c-992a-357ac525965f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:07:53.703266 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 958/1893] 10.222.0.50 () {58 vars in 958 bytes} [Tue Sep 13 16:07:53 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 13 16:07:53.712862 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8791a1a2-2e78-4c21-9ba5-828f806dcc8a None None] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:07:53.714224 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-8791a1a2-2e78-4c21-9ba5-828f806dcc8a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:07:53.724489 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:07:53.724784 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0d233e7-2e50-4c4c-992a-357ac525965f None None] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:07:53.775555 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8791a1a2-2e78-4c21-9ba5-828f806dcc8a None None] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:07:53.777533 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f077972e-1bc0-4ed5-95f1-af4f3e44b916 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f436f07939d4855a21b195bc7566244', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:07:53.778178 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f077972e-1bc0-4ed5-95f1-af4f3e44b916 service neutron] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:07:53.778463 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f077972e-1bc0-4ed5-95f1-af4f3e44b916 service neutron] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:07:53.778892 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f077972e-1bc0-4ed5-95f1-af4f3e44b916 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:07:53.779982 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ba10d44-2fe7-4d6b-85d3-d6f72dff1162 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f436f07939d4855a21b195bc7566244', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:07:53.780602 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ba10d44-2fe7-4d6b-85d3-d6f72dff1162 service neutron] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:07:53.780917 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ba10d44-2fe7-4d6b-85d3-d6f72dff1162 service neutron] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:07:53.781196 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ba10d44-2fe7-4d6b-85d3-d6f72dff1162 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:07:53.788365 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f077972e-1bc0-4ed5-95f1-af4f3e44b916 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:07:53.790473 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f077972e-1bc0-4ed5-95f1-af4f3e44b916 service neutron] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:07:53.792087 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ba10d44-2fe7-4d6b-85d3-d6f72dff1162 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:07:53.793521 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ba10d44-2fe7-4d6b-85d3-d6f72dff1162 service neutron] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:07:53.800104 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 936/1894] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 16:07:53 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 139 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 13 16:07:53.803468 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 959/1895] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 16:07:53 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 91 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 13 16:07:54.213203 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f077972e-1bc0-4ed5-95f1-af4f3e44b916 service neutron] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:07:54.215268 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-f077972e-1bc0-4ed5-95f1-af4f3e44b916 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:07:54.229501 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f077972e-1bc0-4ed5-95f1-af4f3e44b916 service neutron] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:07:54.235057 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-215f1412-6110-4c00-9996-f26e0223b747 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:07:54.236332 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-215f1412-6110-4c00-9996-f26e0223b747 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:07:54.236880 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-215f1412-6110-4c00-9996-f26e0223b747 admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:07:54.237359 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-215f1412-6110-4c00-9996-f26e0223b747 admin admin] PATH_INFO: `/v3/projects/8cb3078cf06341a2a6d49dd368db6eb8` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:07:54.238707 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-215f1412-6110-4c00-9996-f26e0223b747 admin admin] RBAC: Authorizing `identity:delete_project(project_id=8cb3078cf06341a2a6d49dd368db6eb8)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:07:54.242539 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-215f1412-6110-4c00-9996-f26e0223b747 admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:07:54.244134 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ba10d44-2fe7-4d6b-85d3-d6f72dff1162 service neutron] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:07:54.245827 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-4ba10d44-2fe7-4d6b-85d3-d6f72dff1162 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:07:54.260576 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ba10d44-2fe7-4d6b-85d3-d6f72dff1162 service neutron] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:07:54.266705 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f8f00ae-100a-4c26-9035-4ec5347e62e7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:07:54.268029 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f8f00ae-100a-4c26-9035-4ec5347e62e7 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:07:54.268316 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f8f00ae-100a-4c26-9035-4ec5347e62e7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:07:54.268588 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f8f00ae-100a-4c26-9035-4ec5347e62e7 admin admin] PATH_INFO: `/v3/projects/8c14445158b3411bb08c888abafa14cd` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:07:54.269786 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f8f00ae-100a-4c26-9035-4ec5347e62e7 admin admin] RBAC: Authorizing `identity:delete_project(project_id=8c14445158b3411bb08c888abafa14cd)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:07:54.271761 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f8f00ae-100a-4c26-9035-4ec5347e62e7 admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:07:54.301449 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-215f1412-6110-4c00-9996-f26e0223b747 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '8cb3078cf06341a2a6d49dd368db6eb8', 'request_id': 'req-215f1412-6110-4c00-9996-f26e0223b747'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:07:54.313096 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-215f1412-6110-4c00-9996-f26e0223b747 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '8cb3078cf06341a2a6d49dd368db6eb8', 'request_id': 'req-215f1412-6110-4c00-9996-f26e0223b747'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:07:54.321556 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-215f1412-6110-4c00-9996-f26e0223b747 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '8cb3078cf06341a2a6d49dd368db6eb8', 'request_id': 'req-215f1412-6110-4c00-9996-f26e0223b747'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:07:54.325383 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:07:54.340755 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-215f1412-6110-4c00-9996-f26e0223b747 admin admin] The token cache is being invalidate because project 8cb3078cf06341a2a6d49dd368db6eb8 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81438) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 13 16:07:54.341298 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-215f1412-6110-4c00-9996-f26e0223b747 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:07:54.342307 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-215f1412-6110-4c00-9996-f26e0223b747 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:07:54.342826 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-0f8f00ae-100a-4c26-9035-4ec5347e62e7 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '8c14445158b3411bb08c888abafa14cd', 'request_id': 'req-0f8f00ae-100a-4c26-9035-4ec5347e62e7'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:07:54.344350 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 937/1896] 10.222.0.50 () {58 vars in 1233 bytes} [Tue Sep 13 16:07:54 2022] DELETE /identity/v3/projects/8cb3078cf06341a2a6d49dd368db6eb8 => generated 0 bytes in 132 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 16:07:54.348263 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-215f1412-6110-4c00-9996-f26e0223b747 admin admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:07:54.349840 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-215f1412-6110-4c00-9996-f26e0223b747 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:07:54.352893 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-0f8f00ae-100a-4c26-9035-4ec5347e62e7 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '8c14445158b3411bb08c888abafa14cd', 'request_id': 'req-0f8f00ae-100a-4c26-9035-4ec5347e62e7'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:07:54.360391 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-0f8f00ae-100a-4c26-9035-4ec5347e62e7 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '8c14445158b3411bb08c888abafa14cd', 'request_id': 'req-0f8f00ae-100a-4c26-9035-4ec5347e62e7'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:07:54.373527 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-0f8f00ae-100a-4c26-9035-4ec5347e62e7 admin admin] The token cache is being invalidate because project 8c14445158b3411bb08c888abafa14cd was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81437) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 13 16:07:54.374101 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-0f8f00ae-100a-4c26-9035-4ec5347e62e7 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:07:54.375623 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-0f8f00ae-100a-4c26-9035-4ec5347e62e7 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:07:54.377443 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 960/1897] 10.222.0.50 () {58 vars in 1233 bytes} [Tue Sep 13 16:07:54 2022] DELETE /identity/v3/projects/8c14445158b3411bb08c888abafa14cd => generated 0 bytes in 134 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 16:07:54.381068 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f8f00ae-100a-4c26-9035-4ec5347e62e7 admin admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:07:54.383361 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-0f8f00ae-100a-4c26-9035-4ec5347e62e7 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:07:54.426201 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-215f1412-6110-4c00-9996-f26e0223b747 admin admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:07:54.472214 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f8f00ae-100a-4c26-9035-4ec5347e62e7 admin admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:07:54.477956 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec7dcf7d-ef83-4cd8-b729-060e4da37257 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:07:54.478784 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec7dcf7d-ef83-4cd8-b729-060e4da37257 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:07:54.479262 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec7dcf7d-ef83-4cd8-b729-060e4da37257 admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:07:54.479990 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec7dcf7d-ef83-4cd8-b729-060e4da37257 admin admin] PATH_INFO: `/v3/users/c29e087c21ba49d5ae3e5031fd500fda` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:07:54.482734 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec7dcf7d-ef83-4cd8-b729-060e4da37257 admin admin] RBAC: Authorizing `identity:delete_user(user_id=c29e087c21ba49d5ae3e5031fd500fda)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:07:54.485396 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec7dcf7d-ef83-4cd8-b729-060e4da37257 admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:07:54.543511 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-559a2116-150b-41da-af37-c302a9728fcb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:07:54.544175 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-559a2116-150b-41da-af37-c302a9728fcb admin admin] REQUEST_METHOD: `DELETE` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:07:54.544448 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-559a2116-150b-41da-af37-c302a9728fcb admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:07:54.544726 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-559a2116-150b-41da-af37-c302a9728fcb admin admin] PATH_INFO: `/v3/users/c5c76a4638f140739e0d19e303806a9d` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:07:54.547020 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-559a2116-150b-41da-af37-c302a9728fcb admin admin] RBAC: Authorizing `identity:delete_user(user_id=c5c76a4638f140739e0d19e303806a9d)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:07:54.549553 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-559a2116-150b-41da-af37-c302a9728fcb admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:07:54.559878 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=87,keystone:DELETE=12,keystone:INSERT=1 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:07:54.578477 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-ec7dcf7d-ef83-4cd8-b729-060e4da37257 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'c29e087c21ba49d5ae3e5031fd500fda'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:07:54.590327 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-ec7dcf7d-ef83-4cd8-b729-060e4da37257 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'c29e087c21ba49d5ae3e5031fd500fda'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:07:54.609889 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-ec7dcf7d-ef83-4cd8-b729-060e4da37257 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'c29e087c21ba49d5ae3e5031fd500fda'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:07:54.611055 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-ec7dcf7d-ef83-4cd8-b729-060e4da37257 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'c29e087c21ba49d5ae3e5031fd500fda'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:07:54.624132 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-ec7dcf7d-ef83-4cd8-b729-060e4da37257 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'c29e087c21ba49d5ae3e5031fd500fda'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:07:54.633898 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-559a2116-150b-41da-af37-c302a9728fcb admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'c5c76a4638f140739e0d19e303806a9d'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:07:54.635633 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 961/1898] 10.222.0.50 () {58 vars in 1224 bytes} [Tue Sep 13 16:07:54 2022] DELETE /identity/v3/users/c29e087c21ba49d5ae3e5031fd500fda => generated 0 bytes in 255 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 13 16:07:54.643465 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-559a2116-150b-41da-af37-c302a9728fcb admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'c5c76a4638f140739e0d19e303806a9d'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:07:54.669507 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-559a2116-150b-41da-af37-c302a9728fcb admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'c5c76a4638f140739e0d19e303806a9d'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:07:54.670484 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-559a2116-150b-41da-af37-c302a9728fcb admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'c5c76a4638f140739e0d19e303806a9d'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:07:54.680993 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-559a2116-150b-41da-af37-c302a9728fcb admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'c5c76a4638f140739e0d19e303806a9d'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:07:54.692084 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 938/1899] 10.222.0.50 () {58 vars in 1224 bytes} [Tue Sep 13 16:07:54 2022] DELETE /identity/v3/users/c5c76a4638f140739e0d19e303806a9d => generated 0 bytes in 341 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 16:07:55.035022 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec7dcf7d-ef83-4cd8-b729-060e4da37257 admin admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:07:55.036777 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-ec7dcf7d-ef83-4cd8-b729-060e4da37257 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:07:55.064157 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-559a2116-150b-41da-af37-c302a9728fcb admin admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:07:55.066040 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-559a2116-150b-41da-af37-c302a9728fcb admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:07:55.100487 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:07:55.146928 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec7dcf7d-ef83-4cd8-b729-060e4da37257 admin admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:07:55.223126 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-559a2116-150b-41da-af37-c302a9728fcb admin admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:07:55.229155 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1105f99b-1546-44fb-abc9-406cc46fb64a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:07:55.229979 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1105f99b-1546-44fb-abc9-406cc46fb64a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:07:55.230613 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1105f99b-1546-44fb-abc9-406cc46fb64a admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:07:55.231274 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1105f99b-1546-44fb-abc9-406cc46fb64a admin admin] PATH_INFO: `/v3/projects/afd9a3784c544737b692986b19f13247` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:07:55.233139 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1105f99b-1546-44fb-abc9-406cc46fb64a admin admin] RBAC: Authorizing `identity:delete_project(project_id=afd9a3784c544737b692986b19f13247)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:07:55.235486 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1105f99b-1546-44fb-abc9-406cc46fb64a admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:07:55.239920 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-315c812c-0832-473b-a511-7ddf3a5b957f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:07:55.240577 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-315c812c-0832-473b-a511-7ddf3a5b957f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:07:55.240858 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-315c812c-0832-473b-a511-7ddf3a5b957f admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:07:55.242165 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-315c812c-0832-473b-a511-7ddf3a5b957f admin admin] PATH_INFO: `/v3/projects/a9165453186c46ae9d9a80b7aace88b2` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:07:55.246431 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-315c812c-0832-473b-a511-7ddf3a5b957f admin admin] RBAC: Authorizing `identity:delete_project(project_id=a9165453186c46ae9d9a80b7aace88b2)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:07:55.248482 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-315c812c-0832-473b-a511-7ddf3a5b957f admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:07:55.300685 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-1105f99b-1546-44fb-abc9-406cc46fb64a admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'afd9a3784c544737b692986b19f13247', 'request_id': 'req-1105f99b-1546-44fb-abc9-406cc46fb64a'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:07:55.310571 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-1105f99b-1546-44fb-abc9-406cc46fb64a admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'afd9a3784c544737b692986b19f13247', 'request_id': 'req-1105f99b-1546-44fb-abc9-406cc46fb64a'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:07:55.318841 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-1105f99b-1546-44fb-abc9-406cc46fb64a admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'afd9a3784c544737b692986b19f13247', 'request_id': 'req-1105f99b-1546-44fb-abc9-406cc46fb64a'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:07:55.329569 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-315c812c-0832-473b-a511-7ddf3a5b957f admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'a9165453186c46ae9d9a80b7aace88b2', 'request_id': 'req-315c812c-0832-473b-a511-7ddf3a5b957f'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:07:55.333964 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-1105f99b-1546-44fb-abc9-406cc46fb64a admin admin] The token cache is being invalidate because project afd9a3784c544737b692986b19f13247 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81438) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 13 16:07:55.334374 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-1105f99b-1546-44fb-abc9-406cc46fb64a admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:07:55.335726 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-1105f99b-1546-44fb-abc9-406cc46fb64a admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:07:55.337710 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 939/1900] 10.222.0.50 () {58 vars in 1233 bytes} [Tue Sep 13 16:07:55 2022] DELETE /identity/v3/projects/afd9a3784c544737b692986b19f13247 => generated 0 bytes in 274 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 16:07:55.339303 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-315c812c-0832-473b-a511-7ddf3a5b957f admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'a9165453186c46ae9d9a80b7aace88b2', 'request_id': 'req-315c812c-0832-473b-a511-7ddf3a5b957f'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:07:55.342566 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1105f99b-1546-44fb-abc9-406cc46fb64a admin admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:07:55.344280 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-1105f99b-1546-44fb-abc9-406cc46fb64a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:07:55.348978 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-315c812c-0832-473b-a511-7ddf3a5b957f admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'a9165453186c46ae9d9a80b7aace88b2', 'request_id': 'req-315c812c-0832-473b-a511-7ddf3a5b957f'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:07:55.350039 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=88,keystone:DELETE=10,keystone:INSERT=2 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:07:55.371820 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-315c812c-0832-473b-a511-7ddf3a5b957f admin admin] The token cache is being invalidate because project a9165453186c46ae9d9a80b7aace88b2 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81437) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 13 16:07:55.372117 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-315c812c-0832-473b-a511-7ddf3a5b957f admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:07:55.374132 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-315c812c-0832-473b-a511-7ddf3a5b957f admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:07:55.377680 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 962/1901] 10.222.0.50 () {58 vars in 1233 bytes} [Tue Sep 13 16:07:55 2022] DELETE /identity/v3/projects/a9165453186c46ae9d9a80b7aace88b2 => generated 0 bytes in 343 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 13 16:07:55.384714 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8285e565-b2d0-4fb1-9dd5-766e53eb68cf None None] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:07:55.385012 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8285e565-b2d0-4fb1-9dd5-766e53eb68cf None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:07:55.385294 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8285e565-b2d0-4fb1-9dd5-766e53eb68cf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:07:55.413927 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.auth.core [None req-8285e565-b2d0-4fb1-9dd5-766e53eb68cf None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81437) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:07:55.415580 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-8285e565-b2d0-4fb1-9dd5-766e53eb68cf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:07:55.449615 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1105f99b-1546-44fb-abc9-406cc46fb64a admin admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:07:55.496794 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 963/1902] 10.222.0.50 () {58 vars in 958 bytes} [Tue Sep 13 16:07:55 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:07:55.504953 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b83dcacf-e2cc-4fed-96ec-60f3af1a2364 None None] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:07:55.505260 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b83dcacf-e2cc-4fed-96ec-60f3af1a2364 None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:07:55.505610 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b83dcacf-e2cc-4fed-96ec-60f3af1a2364 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:07:55.513647 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:07:55.520635 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7ecfac7-d23f-4150-b5ab-0803181d977c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:07:55.521643 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7ecfac7-d23f-4150-b5ab-0803181d977c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:07:55.521957 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7ecfac7-d23f-4150-b5ab-0803181d977c admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:07:55.522192 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7ecfac7-d23f-4150-b5ab-0803181d977c admin admin] PATH_INFO: `/v3/users/e20033f63de244b4a3e8bf97bfb1404a` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:07:55.524749 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7ecfac7-d23f-4150-b5ab-0803181d977c admin admin] RBAC: Authorizing `identity:delete_user(user_id=e20033f63de244b4a3e8bf97bfb1404a)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:07:55.527333 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7ecfac7-d23f-4150-b5ab-0803181d977c admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:07:55.530885 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.auth.core [None req-b83dcacf-e2cc-4fed-96ec-60f3af1a2364 None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81437) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:07:55.532132 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-b83dcacf-e2cc-4fed-96ec-60f3af1a2364 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:07:55.546132 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 964/1903] 10.222.0.50 () {58 vars in 958 bytes} [Tue Sep 13 16:07:55 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 42 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:07:55.550286 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b83dcacf-e2cc-4fed-96ec-60f3af1a2364 None None] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:07:55.561184 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b83dcacf-e2cc-4fed-96ec-60f3af1a2364 None None] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:07:55.567343 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c186097-b904-4c4d-ac0a-63b1aa55235a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:07:55.568037 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c186097-b904-4c4d-ac0a-63b1aa55235a admin admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:07:55.568398 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c186097-b904-4c4d-ac0a-63b1aa55235a admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:07:55.568764 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c186097-b904-4c4d-ac0a-63b1aa55235a admin admin] PATH_INFO: `/v3/domains` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:07:55.570118 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c186097-b904-4c4d-ac0a-63b1aa55235a admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81437) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 16:07:55.570502 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c186097-b904-4c4d-ac0a-63b1aa55235a admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:07:55.572246 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c186097-b904-4c4d-ac0a-63b1aa55235a admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:07:55.583865 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-c7ecfac7-d23f-4150-b5ab-0803181d977c admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'e20033f63de244b4a3e8bf97bfb1404a'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:07:55.585135 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 965/1904] 10.222.0.50 () {58 vars in 1153 bytes} [Tue Sep 13 16:07:55 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 13 16:07:55.589181 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c186097-b904-4c4d-ac0a-63b1aa55235a admin admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:07:55.591954 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-c7ecfac7-d23f-4150-b5ab-0803181d977c admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'e20033f63de244b4a3e8bf97bfb1404a'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:07:55.598987 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c186097-b904-4c4d-ac0a-63b1aa55235a admin admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:07:55.603758 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4d25481-23bb-495f-a341-a01f86af30ff admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:07:55.604370 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4d25481-23bb-495f-a341-a01f86af30ff admin admin] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:07:55.604637 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4d25481-23bb-495f-a341-a01f86af30ff admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:07:55.604810 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-c7ecfac7-d23f-4150-b5ab-0803181d977c admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'e20033f63de244b4a3e8bf97bfb1404a'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:07:55.604924 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4d25481-23bb-495f-a341-a01f86af30ff admin admin] PATH_INFO: `/v3/projects` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:07:55.605518 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4d25481-23bb-495f-a341-a01f86af30ff admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:07:55.605625 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-c7ecfac7-d23f-4150-b5ab-0803181d977c admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'e20033f63de244b4a3e8bf97bfb1404a'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:07:55.607452 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4d25481-23bb-495f-a341-a01f86af30ff admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:07:55.612734 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-c7ecfac7-d23f-4150-b5ab-0803181d977c admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'e20033f63de244b4a3e8bf97bfb1404a'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:07:55.619527 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 940/1905] 10.222.0.50 () {58 vars in 1224 bytes} [Tue Sep 13 16:07:55 2022] DELETE /identity/v3/users/e20033f63de244b4a3e8bf97bfb1404a => generated 0 bytes in 278 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 16:07:55.638936 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 966/1906] 10.222.0.50 () {60 vars in 1153 bytes} [Tue Sep 13 16:07:55 2022] POST /identity/v3/projects => generated 381 bytes in 48 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 16:07:55.640994 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7ecfac7-d23f-4150-b5ab-0803181d977c admin admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:07:55.642515 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-c7ecfac7-d23f-4150-b5ab-0803181d977c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:07:55.708833 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7ecfac7-d23f-4150-b5ab-0803181d977c admin admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:07:55.790561 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d58cffaa-4feb-4014-af98-1e8f02e61ef7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:07:55.791237 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d58cffaa-4feb-4014-af98-1e8f02e61ef7 admin admin] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:07:55.791807 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d58cffaa-4feb-4014-af98-1e8f02e61ef7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:07:55.792089 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d58cffaa-4feb-4014-af98-1e8f02e61ef7 admin admin] PATH_INFO: `/v3/users` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:07:55.793414 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d58cffaa-4feb-4014-af98-1e8f02e61ef7 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:07:55.796302 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d58cffaa-4feb-4014-af98-1e8f02e61ef7 admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:07:55.855226 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4d25481-23bb-495f-a341-a01f86af30ff admin admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:07:55.855985 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 941/1907] 10.222.0.50 () {60 vars in 1144 bytes} [Tue Sep 13 16:07:55 2022] POST /identity/v3/users => generated 291 bytes in 215 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 16:07:55.878730 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4d25481-23bb-495f-a341-a01f86af30ff admin admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:07:55.894048 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1b826d0-dd4c-4b3e-a740-73e97a5aa69c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:07:55.895162 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1b826d0-dd4c-4b3e-a740-73e97a5aa69c admin admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:07:55.896402 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1b826d0-dd4c-4b3e-a740-73e97a5aa69c admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:07:55.897081 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1b826d0-dd4c-4b3e-a740-73e97a5aa69c admin admin] PATH_INFO: `/v3/roles` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:07:55.898282 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1b826d0-dd4c-4b3e-a740-73e97a5aa69c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:07:55.902012 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1b826d0-dd4c-4b3e-a740-73e97a5aa69c admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:07:55.925398 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 967/1908] 10.222.0.50 () {58 vars in 1122 bytes} [Tue Sep 13 16:07:55 2022] GET /identity/v3/roles => generated 1420 bytes in 71 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 16:07:55.929857 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d58cffaa-4feb-4014-af98-1e8f02e61ef7 admin admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:07:55.940608 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d58cffaa-4feb-4014-af98-1e8f02e61ef7 admin admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:07:55.946363 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7c94069-e1ea-47ec-a7e1-9a76c9ff541c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:07:55.946848 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7c94069-e1ea-47ec-a7e1-9a76c9ff541c admin admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:07:55.947285 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7c94069-e1ea-47ec-a7e1-9a76c9ff541c admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:07:55.947545 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7c94069-e1ea-47ec-a7e1-9a76c9ff541c admin admin] PATH_INFO: `/v3/roles` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:07:55.948197 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7c94069-e1ea-47ec-a7e1-9a76c9ff541c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:07:55.949858 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7c94069-e1ea-47ec-a7e1-9a76c9ff541c admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:07:55.966866 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1b826d0-dd4c-4b3e-a740-73e97a5aa69c admin admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:07:55.973191 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 942/1909] 10.222.0.50 () {58 vars in 1122 bytes} [Tue Sep 13 16:07:55 2022] GET /identity/v3/roles => generated 1420 bytes in 44 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 16:07:55.998552 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1b826d0-dd4c-4b3e-a740-73e97a5aa69c admin admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:07:56.009300 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebda2dc1-e778-4650-8675-fa6fcf9dbc6f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:07:56.010335 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebda2dc1-e778-4650-8675-fa6fcf9dbc6f admin admin] REQUEST_METHOD: `PUT` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:07:56.010649 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebda2dc1-e778-4650-8675-fa6fcf9dbc6f admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:07:56.011005 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebda2dc1-e778-4650-8675-fa6fcf9dbc6f admin admin] PATH_INFO: `/v3/projects/3d03640f9cfc4870919231b4821b7d3a/users/469a7984fd0440edb1c0058b7fbdb3e2/roles/15db4aaca3264ae29bde38903ba43471` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:07:56.033600 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebda2dc1-e778-4650-8675-fa6fcf9dbc6f admin admin] RBAC: Authorizing `identity:create_grant(project_id=3d03640f9cfc4870919231b4821b7d3a, user_id=469a7984fd0440edb1c0058b7fbdb3e2, role_id=15db4aaca3264ae29bde38903ba43471)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:07:56.037249 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebda2dc1-e778-4650-8675-fa6fcf9dbc6f admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:07:56.072524 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70579a5b-6f4e-4329-8d3d-c8ba9dae1722 None None] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:07:56.076213 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70579a5b-6f4e-4329-8d3d-c8ba9dae1722 None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:07:56.076784 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70579a5b-6f4e-4329-8d3d-c8ba9dae1722 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:07:56.078747 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 968/1910] 10.222.0.50 () {60 vars in 1483 bytes} [Tue Sep 13 16:07:55 2022] PUT /identity/v3/projects/3d03640f9cfc4870919231b4821b7d3a/users/469a7984fd0440edb1c0058b7fbdb3e2/roles/15db4aaca3264ae29bde38903ba43471 => generated 0 bytes in 91 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 16:07:56.089381 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=86,keystone:DELETE=10,keystone:INSERT=4 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:07:56.098800 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebda2dc1-e778-4650-8675-fa6fcf9dbc6f admin admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:07:56.102337 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-ebda2dc1-e778-4650-8675-fa6fcf9dbc6f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:07:56.135772 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.auth.core [None req-70579a5b-6f4e-4329-8d3d-c8ba9dae1722 None None] MFA Rules not processed for user `469a7984fd0440edb1c0058b7fbdb3e2`. Rule list: `[]` (Enabled: `True`). {{(pid=81438) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:07:56.137355 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-70579a5b-6f4e-4329-8d3d-c8ba9dae1722 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:07:56.249093 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebda2dc1-e778-4650-8675-fa6fcf9dbc6f admin admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:07:56.254493 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbe33a7f-825d-486f-91d8-ace19cf5d3ca admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:07:56.259781 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 943/1911] 10.222.0.50 () {58 vars in 958 bytes} [Tue Sep 13 16:07:56 2022] POST /identity/v3/auth/tokens => generated 3258 bytes in 185 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:07:56.259946 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbe33a7f-825d-486f-91d8-ace19cf5d3ca admin admin] REQUEST_METHOD: `DELETE` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:07:56.259946 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbe33a7f-825d-486f-91d8-ace19cf5d3ca admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:07:56.259946 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbe33a7f-825d-486f-91d8-ace19cf5d3ca admin admin] PATH_INFO: `/v3/projects/4b57870b0d4e416f8ea06c6a1683a10f` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:07:56.259946 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbe33a7f-825d-486f-91d8-ace19cf5d3ca admin admin] RBAC: Authorizing `identity:delete_project(project_id=4b57870b0d4e416f8ea06c6a1683a10f)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:07:56.260248 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbe33a7f-825d-486f-91d8-ace19cf5d3ca admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:07:56.272259 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70579a5b-6f4e-4329-8d3d-c8ba9dae1722 None None] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:07:56.284044 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:DELETE=2,keystone:SELECT=96,keystone:INSERT=2 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:07:56.295887 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70579a5b-6f4e-4329-8d3d-c8ba9dae1722 None None] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:07:56.306833 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf87471b-ae07-4967-b272-5f4e4ea3cc37 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:07:56.307814 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf87471b-ae07-4967-b272-5f4e4ea3cc37 admin admin] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:07:56.308111 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf87471b-ae07-4967-b272-5f4e4ea3cc37 admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:07:56.309001 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf87471b-ae07-4967-b272-5f4e4ea3cc37 admin admin] PATH_INFO: `/v3/projects` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:07:56.310064 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf87471b-ae07-4967-b272-5f4e4ea3cc37 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:07:56.314378 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf87471b-ae07-4967-b272-5f4e4ea3cc37 admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:07:56.337939 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-dbe33a7f-825d-486f-91d8-ace19cf5d3ca admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '4b57870b0d4e416f8ea06c6a1683a10f', 'request_id': 'req-dbe33a7f-825d-486f-91d8-ace19cf5d3ca'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:07:56.355336 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-dbe33a7f-825d-486f-91d8-ace19cf5d3ca admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '4b57870b0d4e416f8ea06c6a1683a10f', 'request_id': 'req-dbe33a7f-825d-486f-91d8-ace19cf5d3ca'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:07:56.360831 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 944/1912] 10.222.0.50 () {60 vars in 1153 bytes} [Tue Sep 13 16:07:56 2022] POST /identity/v3/projects => generated 381 bytes in 91 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 16:07:56.365367 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf87471b-ae07-4967-b272-5f4e4ea3cc37 admin admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:07:56.373515 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-dbe33a7f-825d-486f-91d8-ace19cf5d3ca admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '4b57870b0d4e416f8ea06c6a1683a10f', 'request_id': 'req-dbe33a7f-825d-486f-91d8-ace19cf5d3ca'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:07:56.396213 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-dbe33a7f-825d-486f-91d8-ace19cf5d3ca admin admin] The token cache is being invalidate because project 4b57870b0d4e416f8ea06c6a1683a10f was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81437) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 13 16:07:56.396571 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-dbe33a7f-825d-486f-91d8-ace19cf5d3ca admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:07:56.397319 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-dbe33a7f-825d-486f-91d8-ace19cf5d3ca admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:07:56.399336 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 969/1913] 10.222.0.50 () {58 vars in 1233 bytes} [Tue Sep 13 16:07:56 2022] DELETE /identity/v3/projects/4b57870b0d4e416f8ea06c6a1683a10f => generated 0 bytes in 301 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 13 16:07:56.419745 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f69c424-48ed-4f63-bed1-58dbb70ff835 None None] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:07:56.422423 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f69c424-48ed-4f63-bed1-58dbb70ff835 None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:07:56.422850 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f69c424-48ed-4f63-bed1-58dbb70ff835 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:07:56.462002 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.auth.core [None req-8f69c424-48ed-4f63-bed1-58dbb70ff835 None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81437) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:07:56.464724 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-8f69c424-48ed-4f63-bed1-58dbb70ff835 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:07:56.475221 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf87471b-ae07-4967-b272-5f4e4ea3cc37 admin admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:07:56.482540 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f64e5d2b-8284-4478-932e-e422ba739edd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:07:56.483468 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f64e5d2b-8284-4478-932e-e422ba739edd admin admin] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:07:56.483980 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f64e5d2b-8284-4478-932e-e422ba739edd admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:07:56.484556 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f64e5d2b-8284-4478-932e-e422ba739edd admin admin] PATH_INFO: `/v3/users` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:07:56.485229 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f64e5d2b-8284-4478-932e-e422ba739edd admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:07:56.487429 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f64e5d2b-8284-4478-932e-e422ba739edd admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:07:56.512741 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 970/1914] 10.222.0.50 () {58 vars in 958 bytes} [Tue Sep 13 16:07:56 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:07:56.544347 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b3a8763-2c82-4e91-98ca-8c7ec664ad35 None None] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:07:56.544886 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b3a8763-2c82-4e91-98ca-8c7ec664ad35 None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:07:56.545349 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b3a8763-2c82-4e91-98ca-8c7ec664ad35 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:07:56.547972 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 945/1915] 10.222.0.50 () {60 vars in 1144 bytes} [Tue Sep 13 16:07:56 2022] POST /identity/v3/users => generated 297 bytes in 183 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 16:07:56.552648 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f64e5d2b-8284-4478-932e-e422ba739edd admin admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:07:56.554670 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-f64e5d2b-8284-4478-932e-e422ba739edd admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:07:56.578496 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f64e5d2b-8284-4478-932e-e422ba739edd admin admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:07:56.579401 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.auth.core [None req-1b3a8763-2c82-4e91-98ca-8c7ec664ad35 None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81437) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:07:56.581611 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-1b3a8763-2c82-4e91-98ca-8c7ec664ad35 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:07:56.587232 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-485238eb-8834-4170-8979-16b61d0435bd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:07:56.588383 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-485238eb-8834-4170-8979-16b61d0435bd admin admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:07:56.588669 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-485238eb-8834-4170-8979-16b61d0435bd admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:07:56.589088 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-485238eb-8834-4170-8979-16b61d0435bd admin admin] PATH_INFO: `/v3/roles` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:07:56.589943 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-485238eb-8834-4170-8979-16b61d0435bd admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:07:56.591619 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-485238eb-8834-4170-8979-16b61d0435bd admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:07:56.598239 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 971/1916] 10.222.0.50 () {58 vars in 958 bytes} [Tue Sep 13 16:07:56 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 74 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:07:56.599777 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b3a8763-2c82-4e91-98ca-8c7ec664ad35 None None] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:07:56.603641 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 946/1917] 10.222.0.50 () {58 vars in 1122 bytes} [Tue Sep 13 16:07:56 2022] GET /identity/v3/roles => generated 1420 bytes in 51 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 16:07:56.607857 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-485238eb-8834-4170-8979-16b61d0435bd admin admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:07:56.610692 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b3a8763-2c82-4e91-98ca-8c7ec664ad35 None None] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:07:56.618437 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a0edb00-9227-4358-9b0c-c523630ff58b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:07:56.619674 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a0edb00-9227-4358-9b0c-c523630ff58b admin admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:07:56.620033 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a0edb00-9227-4358-9b0c-c523630ff58b admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:07:56.620302 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-485238eb-8834-4170-8979-16b61d0435bd admin admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:07:56.620425 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a0edb00-9227-4358-9b0c-c523630ff58b admin admin] PATH_INFO: `/v3/domains` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:07:56.621288 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a0edb00-9227-4358-9b0c-c523630ff58b admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81437) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 16:07:56.621531 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a0edb00-9227-4358-9b0c-c523630ff58b admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:07:56.623575 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a0edb00-9227-4358-9b0c-c523630ff58b admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:07:56.626206 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a818fe31-5fc0-46fb-888d-d35b9f91d555 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:07:56.626392 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a818fe31-5fc0-46fb-888d-d35b9f91d555 admin admin] REQUEST_METHOD: `PUT` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:07:56.626604 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a818fe31-5fc0-46fb-888d-d35b9f91d555 admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:07:56.626882 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a818fe31-5fc0-46fb-888d-d35b9f91d555 admin admin] PATH_INFO: `/v3/projects/1e8eb6a8bffd4e8286ed9fd76f77dd92/users/2f6e7bbfe4d74542aea3f50aa23353bf/roles/031d4e1a23e34f71858fc4a6dfbd4ccb` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:07:56.642987 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 972/1918] 10.222.0.50 () {58 vars in 1153 bytes} [Tue Sep 13 16:07:56 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 41 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 16:07:56.645408 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a0edb00-9227-4358-9b0c-c523630ff58b admin admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:07:56.650617 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a818fe31-5fc0-46fb-888d-d35b9f91d555 admin admin] RBAC: Authorizing `identity:create_grant(project_id=1e8eb6a8bffd4e8286ed9fd76f77dd92, user_id=2f6e7bbfe4d74542aea3f50aa23353bf, role_id=031d4e1a23e34f71858fc4a6dfbd4ccb)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:07:56.653748 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a818fe31-5fc0-46fb-888d-d35b9f91d555 admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:07:56.662217 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a0edb00-9227-4358-9b0c-c523630ff58b admin admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:07:56.668809 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 947/1919] 10.222.0.50 () {60 vars in 1483 bytes} [Tue Sep 13 16:07:56 2022] PUT /identity/v3/projects/1e8eb6a8bffd4e8286ed9fd76f77dd92/users/2f6e7bbfe4d74542aea3f50aa23353bf/roles/031d4e1a23e34f71858fc4a6dfbd4ccb => generated 0 bytes in 61 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 13 16:07:56.672828 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a818fe31-5fc0-46fb-888d-d35b9f91d555 admin admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:07:56.707179 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:07:56.748078 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9110535e-9492-48e1-84dd-3310996dc670 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:07:56.748646 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9110535e-9492-48e1-84dd-3310996dc670 admin admin] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:07:56.749012 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9110535e-9492-48e1-84dd-3310996dc670 admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:07:56.749293 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9110535e-9492-48e1-84dd-3310996dc670 admin admin] PATH_INFO: `/v3/projects` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:07:56.750058 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9110535e-9492-48e1-84dd-3310996dc670 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:07:56.752053 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9110535e-9492-48e1-84dd-3310996dc670 admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:07:56.762888 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a818fe31-5fc0-46fb-888d-d35b9f91d555 admin admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:07:56.768568 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f05e35f9-af8b-496b-b22a-c15356483026 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:07:56.769196 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f05e35f9-af8b-496b-b22a-c15356483026 admin admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:07:56.770418 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f05e35f9-af8b-496b-b22a-c15356483026 admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:07:56.770418 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f05e35f9-af8b-496b-b22a-c15356483026 admin admin] PATH_INFO: `/v3/roles` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:07:56.770597 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f05e35f9-af8b-496b-b22a-c15356483026 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:07:56.772925 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f05e35f9-af8b-496b-b22a-c15356483026 admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:07:56.789994 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 948/1920] 10.222.0.50 () {58 vars in 1122 bytes} [Tue Sep 13 16:07:56 2022] GET /identity/v3/roles => generated 1420 bytes in 113 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 13 16:07:56.790374 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f05e35f9-af8b-496b-b22a-c15356483026 admin admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:07:56.796914 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 973/1921] 10.222.0.50 () {60 vars in 1153 bytes} [Tue Sep 13 16:07:56 2022] POST /identity/v3/projects => generated 373 bytes in 153 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 16:07:56.801299 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9110535e-9492-48e1-84dd-3310996dc670 admin admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:07:56.801644 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f05e35f9-af8b-496b-b22a-c15356483026 admin admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:07:56.870538 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65b95117-b4d1-4b43-9626-344299df94c7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:07:56.871255 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65b95117-b4d1-4b43-9626-344299df94c7 admin admin] REQUEST_METHOD: `PUT` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:07:56.871660 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65b95117-b4d1-4b43-9626-344299df94c7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:07:56.871947 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65b95117-b4d1-4b43-9626-344299df94c7 admin admin] PATH_INFO: `/v3/projects/1e8eb6a8bffd4e8286ed9fd76f77dd92/users/2f6e7bbfe4d74542aea3f50aa23353bf/roles/031d4e1a23e34f71858fc4a6dfbd4ccb` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:07:56.874100 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65b95117-b4d1-4b43-9626-344299df94c7 admin admin] RBAC: Authorizing `identity:create_grant(project_id=1e8eb6a8bffd4e8286ed9fd76f77dd92, user_id=2f6e7bbfe4d74542aea3f50aa23353bf, role_id=031d4e1a23e34f71858fc4a6dfbd4ccb)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:07:56.876927 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65b95117-b4d1-4b43-9626-344299df94c7 admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:07:56.879031 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9110535e-9492-48e1-84dd-3310996dc670 admin admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:07:56.885340 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2b99a3a-64a9-49cb-8c52-baa005cfdd26 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:07:56.886034 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2b99a3a-64a9-49cb-8c52-baa005cfdd26 admin admin] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:07:56.886365 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2b99a3a-64a9-49cb-8c52-baa005cfdd26 admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:07:56.887163 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2b99a3a-64a9-49cb-8c52-baa005cfdd26 admin admin] PATH_INFO: `/v3/users` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:07:56.887344 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2b99a3a-64a9-49cb-8c52-baa005cfdd26 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:07:56.889391 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 949/1922] 10.222.0.50 () {60 vars in 1483 bytes} [Tue Sep 13 16:07:56 2022] PUT /identity/v3/projects/1e8eb6a8bffd4e8286ed9fd76f77dd92/users/2f6e7bbfe4d74542aea3f50aa23353bf/roles/031d4e1a23e34f71858fc4a6dfbd4ccb => generated 0 bytes in 99 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 16:07:56.889506 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2b99a3a-64a9-49cb-8c52-baa005cfdd26 admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:07:56.900849 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:07:56.901691 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d3c3b09-ec77-4280-a007-ac6bafa6c71b None None] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:07:56.901986 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d3c3b09-ec77-4280-a007-ac6bafa6c71b None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:07:56.902271 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d3c3b09-ec77-4280-a007-ac6bafa6c71b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:07:56.922817 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 974/1923] 10.222.0.50 () {60 vars in 1144 bytes} [Tue Sep 13 16:07:56 2022] POST /identity/v3/users => generated 287 bytes in 122 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 13 16:07:56.926685 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2b99a3a-64a9-49cb-8c52-baa005cfdd26 admin admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:07:56.945726 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.auth.core [None req-6d3c3b09-ec77-4280-a007-ac6bafa6c71b None None] MFA Rules not processed for user `2f6e7bbfe4d74542aea3f50aa23353bf`. Rule list: `[]` (Enabled: `True`). {{(pid=81438) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:07:56.947493 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2b99a3a-64a9-49cb-8c52-baa005cfdd26 admin admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:07:56.947706 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-6d3c3b09-ec77-4280-a007-ac6bafa6c71b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:07:57.038034 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-986aa820-c630-44da-819e-c7e4143c8877 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:07:57.039371 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-986aa820-c630-44da-819e-c7e4143c8877 admin admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:07:57.039712 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-986aa820-c630-44da-819e-c7e4143c8877 admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:07:57.039992 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-986aa820-c630-44da-819e-c7e4143c8877 admin admin] PATH_INFO: `/v3/roles` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:07:57.040707 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-986aa820-c630-44da-819e-c7e4143c8877 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:07:57.042332 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-986aa820-c630-44da-819e-c7e4143c8877 admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:07:57.051266 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 950/1924] 10.222.0.50 () {58 vars in 958 bytes} [Tue Sep 13 16:07:56 2022] POST /identity/v3/auth/tokens => generated 3325 bytes in 151 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:07:57.062008 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 975/1925] 10.222.0.50 () {58 vars in 1122 bytes} [Tue Sep 13 16:07:56 2022] GET /identity/v3/roles => generated 1420 bytes in 136 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 13 16:07:57.063967 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d3c3b09-ec77-4280-a007-ac6bafa6c71b None None] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:07:57.075042 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-986aa820-c630-44da-819e-c7e4143c8877 admin admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:07:57.075042 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-986aa820-c630-44da-819e-c7e4143c8877 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:07:57.088802 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d3c3b09-ec77-4280-a007-ac6bafa6c71b None None] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:07:57.095349 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f03a84b-ee58-4bc6-952f-20d9eee7b72f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:07:57.098434 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f03a84b-ee58-4bc6-952f-20d9eee7b72f admin admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:07:57.098434 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f03a84b-ee58-4bc6-952f-20d9eee7b72f admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:07:57.098434 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f03a84b-ee58-4bc6-952f-20d9eee7b72f admin admin] PATH_INFO: `/v3/roles` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:07:57.098434 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f03a84b-ee58-4bc6-952f-20d9eee7b72f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:07:57.101364 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f03a84b-ee58-4bc6-952f-20d9eee7b72f admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:07:57.124848 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 951/1926] 10.222.0.50 () {58 vars in 1122 bytes} [Tue Sep 13 16:07:57 2022] GET /identity/v3/roles => generated 1420 bytes in 59 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 16:07:57.127297 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f03a84b-ee58-4bc6-952f-20d9eee7b72f admin admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:07:57.141560 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f03a84b-ee58-4bc6-952f-20d9eee7b72f admin admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:07:57.150226 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8193c558-1c04-4500-867b-39a47e2e1be2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:07:57.151216 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8193c558-1c04-4500-867b-39a47e2e1be2 admin admin] REQUEST_METHOD: `PUT` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:07:57.151797 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8193c558-1c04-4500-867b-39a47e2e1be2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:07:57.152011 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8193c558-1c04-4500-867b-39a47e2e1be2 admin admin] PATH_INFO: `/v3/projects/a19f8d067f67490cac7c827edcaaa8dc/users/96751dfa79154d3885eca3e37648c2a9/roles/15db4aaca3264ae29bde38903ba43471` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:07:57.163488 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-986aa820-c630-44da-819e-c7e4143c8877 admin admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:07:57.171043 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce9abd84-b75b-4653-bb05-53b6d877f625 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7fd6b5afd67b48f9b0f0597ccd4cf4df', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:07:57.171265 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce9abd84-b75b-4653-bb05-53b6d877f625 service cinder] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:07:57.171654 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce9abd84-b75b-4653-bb05-53b6d877f625 service cinder] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:07:57.172028 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce9abd84-b75b-4653-bb05-53b6d877f625 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:07:57.178467 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8193c558-1c04-4500-867b-39a47e2e1be2 admin admin] RBAC: Authorizing `identity:create_grant(project_id=a19f8d067f67490cac7c827edcaaa8dc, user_id=96751dfa79154d3885eca3e37648c2a9, role_id=15db4aaca3264ae29bde38903ba43471)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:07:57.182210 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8193c558-1c04-4500-867b-39a47e2e1be2 admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:07:57.183851 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce9abd84-b75b-4653-bb05-53b6d877f625 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:07:57.185771 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce9abd84-b75b-4653-bb05-53b6d877f625 service cinder] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:07:57.215125 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 976/1928] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 16:07:57 2022] GET /identity/v3/auth/tokens => generated 3325 bytes in 143 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 13 16:07:57.215624 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 952/1928] 10.222.0.50 () {60 vars in 1483 bytes} [Tue Sep 13 16:07:57 2022] PUT /identity/v3/projects/a19f8d067f67490cac7c827edcaaa8dc/users/96751dfa79154d3885eca3e37648c2a9/roles/15db4aaca3264ae29bde38903ba43471 => generated 0 bytes in 78 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 16:07:57.217610 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce69dc5f-c5a5-42db-a9f0-a54bfef0af43 None None] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:07:57.218619 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce69dc5f-c5a5-42db-a9f0-a54bfef0af43 None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:07:57.218619 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce69dc5f-c5a5-42db-a9f0-a54bfef0af43 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:07:57.265838 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.auth.core [None req-ce69dc5f-c5a5-42db-a9f0-a54bfef0af43 None None] MFA Rules not processed for user `96751dfa79154d3885eca3e37648c2a9`. Rule list: `[]` (Enabled: `True`). {{(pid=81437) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:07:57.265838 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-ce69dc5f-c5a5-42db-a9f0-a54bfef0af43 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:07:57.350179 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8193c558-1c04-4500-867b-39a47e2e1be2 admin admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:07:57.351949 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-8193c558-1c04-4500-867b-39a47e2e1be2 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:07:57.359542 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:07:57.375849 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 977/1929] 10.222.0.50 () {58 vars in 958 bytes} [Tue Sep 13 16:07:57 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 159 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:07:57.388507 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce69dc5f-c5a5-42db-a9f0-a54bfef0af43 None None] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:07:57.409209 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:INSERT=2,keystone:SELECT=98 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:07:57.415996 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8193c558-1c04-4500-867b-39a47e2e1be2 admin admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:07:57.419818 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-067e8268-dbd8-4c9c-8c8a-a99399b7f158 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7fd6b5afd67b48f9b0f0597ccd4cf4df', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:07:57.420439 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-067e8268-dbd8-4c9c-8c8a-a99399b7f158 service cinder] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:07:57.420710 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-067e8268-dbd8-4c9c-8c8a-a99399b7f158 service cinder] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:07:57.420986 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-067e8268-dbd8-4c9c-8c8a-a99399b7f158 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:07:57.422714 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-067e8268-dbd8-4c9c-8c8a-a99399b7f158 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:07:57.443299 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce69dc5f-c5a5-42db-a9f0-a54bfef0af43 None None] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:07:57.447798 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5775bbd1-108c-4fd7-b740-da862eb0185f service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7fd6b5afd67b48f9b0f0597ccd4cf4df', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:07:57.448405 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5775bbd1-108c-4fd7-b740-da862eb0185f service cinder] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:07:57.448689 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5775bbd1-108c-4fd7-b740-da862eb0185f service cinder] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:07:57.448964 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5775bbd1-108c-4fd7-b740-da862eb0185f service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:07:57.457348 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5775bbd1-108c-4fd7-b740-da862eb0185f service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:07:57.458777 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5775bbd1-108c-4fd7-b740-da862eb0185f service cinder] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:07:57.468659 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 978/1930] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 16:07:57 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 81 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 13 16:07:57.483075 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-067e8268-dbd8-4c9c-8c8a-a99399b7f158 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:07:57.484777 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-067e8268-dbd8-4c9c-8c8a-a99399b7f158 service cinder] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:07:57.494589 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 953/1931] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 16:07:57 2022] GET /identity/v3/auth/tokens => generated 3258 bytes in 145 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 13 16:07:58.104935 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5775bbd1-108c-4fd7-b740-da862eb0185f service cinder] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:07:58.193405 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5775bbd1-108c-4fd7-b740-da862eb0185f service cinder] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:07:58.199854 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef9df320-501a-480e-b621-ff65cd5e58eb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:07:58.200646 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef9df320-501a-480e-b621-ff65cd5e58eb admin admin] REQUEST_METHOD: `DELETE` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:07:58.201002 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef9df320-501a-480e-b621-ff65cd5e58eb admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:07:58.201325 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef9df320-501a-480e-b621-ff65cd5e58eb admin admin] PATH_INFO: `/v3/users/469a7984fd0440edb1c0058b7fbdb3e2` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:07:58.202647 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef9df320-501a-480e-b621-ff65cd5e58eb admin admin] RBAC: Authorizing `identity:delete_user(user_id=469a7984fd0440edb1c0058b7fbdb3e2)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:07:58.204712 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef9df320-501a-480e-b621-ff65cd5e58eb admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:07:58.264460 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-ef9df320-501a-480e-b621-ff65cd5e58eb admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '469a7984fd0440edb1c0058b7fbdb3e2'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:07:58.273529 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-ef9df320-501a-480e-b621-ff65cd5e58eb admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '469a7984fd0440edb1c0058b7fbdb3e2'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:07:58.302536 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-ef9df320-501a-480e-b621-ff65cd5e58eb admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '469a7984fd0440edb1c0058b7fbdb3e2'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:07:58.303304 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-ef9df320-501a-480e-b621-ff65cd5e58eb admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '469a7984fd0440edb1c0058b7fbdb3e2'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:07:58.310436 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-ef9df320-501a-480e-b621-ff65cd5e58eb admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '469a7984fd0440edb1c0058b7fbdb3e2'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:07:58.316343 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 979/1932] 10.222.0.50 () {58 vars in 1224 bytes} [Tue Sep 13 16:07:58 2022] DELETE /identity/v3/users/469a7984fd0440edb1c0058b7fbdb3e2 => generated 0 bytes in 212 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 16:07:58.321060 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc9c9a1e-5038-4498-8a81-9096514292de None None] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:07:58.321355 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc9c9a1e-5038-4498-8a81-9096514292de None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:07:58.321632 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc9c9a1e-5038-4498-8a81-9096514292de None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:07:58.352047 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.auth.core [None req-dc9c9a1e-5038-4498-8a81-9096514292de None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81438) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:07:58.353260 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-dc9c9a1e-5038-4498-8a81-9096514292de None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:07:58.423775 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 954/1933] 10.222.0.50 () {58 vars in 958 bytes} [Tue Sep 13 16:07:58 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:07:58.434490 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef9df320-501a-480e-b621-ff65cd5e58eb admin admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:07:58.435945 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-ef9df320-501a-480e-b621-ff65cd5e58eb admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:07:58.492326 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:07:58.507331 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef9df320-501a-480e-b621-ff65cd5e58eb admin admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:07:58.512870 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45a6b62b-a498-4d1b-81a6-47b586304346 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f436f07939d4855a21b195bc7566244', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:07:58.513485 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45a6b62b-a498-4d1b-81a6-47b586304346 service neutron] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:07:58.513936 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45a6b62b-a498-4d1b-81a6-47b586304346 service neutron] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:07:58.514215 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45a6b62b-a498-4d1b-81a6-47b586304346 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:07:58.524546 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45a6b62b-a498-4d1b-81a6-47b586304346 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:07:58.526021 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45a6b62b-a498-4d1b-81a6-47b586304346 service neutron] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:07:58.539187 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 980/1934] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 16:07:58 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 105 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 16:07:58.855225 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc9c9a1e-5038-4498-8a81-9096514292de None None] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:07:58.856707 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-dc9c9a1e-5038-4498-8a81-9096514292de None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:07:58.869215 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc9c9a1e-5038-4498-8a81-9096514292de None None] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:07:58.874110 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01c7eb03-6b7a-48b9-9077-1b3494e02f88 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:07:58.874815 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01c7eb03-6b7a-48b9-9077-1b3494e02f88 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:07:58.875284 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01c7eb03-6b7a-48b9-9077-1b3494e02f88 admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:07:58.875571 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01c7eb03-6b7a-48b9-9077-1b3494e02f88 admin admin] PATH_INFO: `/v3/projects/3d03640f9cfc4870919231b4821b7d3a` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:07:58.876714 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01c7eb03-6b7a-48b9-9077-1b3494e02f88 admin admin] RBAC: Authorizing `identity:delete_project(project_id=3d03640f9cfc4870919231b4821b7d3a)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:07:58.879470 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01c7eb03-6b7a-48b9-9077-1b3494e02f88 admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:07:58.931256 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-01c7eb03-6b7a-48b9-9077-1b3494e02f88 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '3d03640f9cfc4870919231b4821b7d3a', 'request_id': 'req-01c7eb03-6b7a-48b9-9077-1b3494e02f88'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:07:58.935025 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=94,keystone:DELETE=5,keystone:INSERT=1 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:07:58.939164 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-01c7eb03-6b7a-48b9-9077-1b3494e02f88 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '3d03640f9cfc4870919231b4821b7d3a', 'request_id': 'req-01c7eb03-6b7a-48b9-9077-1b3494e02f88'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:07:58.942974 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97167383-5536-4e1d-bc84-faa92c1d7385 None None] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:07:58.943625 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97167383-5536-4e1d-bc84-faa92c1d7385 None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:07:58.944934 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97167383-5536-4e1d-bc84-faa92c1d7385 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:07:58.945757 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-01c7eb03-6b7a-48b9-9077-1b3494e02f88 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '3d03640f9cfc4870919231b4821b7d3a', 'request_id': 'req-01c7eb03-6b7a-48b9-9077-1b3494e02f88'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:07:58.958518 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-01c7eb03-6b7a-48b9-9077-1b3494e02f88 admin admin] The token cache is being invalidate because project 3d03640f9cfc4870919231b4821b7d3a was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81438) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 13 16:07:58.958818 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-01c7eb03-6b7a-48b9-9077-1b3494e02f88 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:07:58.959573 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-01c7eb03-6b7a-48b9-9077-1b3494e02f88 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:07:58.961533 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 955/1935] 10.222.0.50 () {58 vars in 1233 bytes} [Tue Sep 13 16:07:58 2022] DELETE /identity/v3/projects/3d03640f9cfc4870919231b4821b7d3a => generated 0 bytes in 107 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 16:07:58.965335 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01c7eb03-6b7a-48b9-9077-1b3494e02f88 admin admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:07:58.966115 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.auth.core [None req-97167383-5536-4e1d-bc84-faa92c1d7385 None None] MFA Rules not processed for user `96751dfa79154d3885eca3e37648c2a9`. Rule list: `[]` (Enabled: `True`). {{(pid=81437) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:07:58.966717 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-01c7eb03-6b7a-48b9-9077-1b3494e02f88 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:07:58.967451 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-97167383-5536-4e1d-bc84-faa92c1d7385 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:07:59.026308 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 981/1936] 10.222.0.50 () {58 vars in 958 bytes} [Tue Sep 13 16:07:58 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 13 16:07:59.050252 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01c7eb03-6b7a-48b9-9077-1b3494e02f88 admin admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:07:59.055909 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e579ff4-3f05-4fe4-b589-2ab741cc34d7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:07:59.056270 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e579ff4-3f05-4fe4-b589-2ab741cc34d7 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:07:59.056581 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e579ff4-3f05-4fe4-b589-2ab741cc34d7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:07:59.056855 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e579ff4-3f05-4fe4-b589-2ab741cc34d7 admin admin] PATH_INFO: `/v3/users/2f6e7bbfe4d74542aea3f50aa23353bf` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:07:59.057991 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e579ff4-3f05-4fe4-b589-2ab741cc34d7 admin admin] RBAC: Authorizing `identity:delete_user(user_id=2f6e7bbfe4d74542aea3f50aa23353bf)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:07:59.059961 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e579ff4-3f05-4fe4-b589-2ab741cc34d7 admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:07:59.111456 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-6e579ff4-3f05-4fe4-b589-2ab741cc34d7 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '2f6e7bbfe4d74542aea3f50aa23353bf'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:07:59.119126 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-6e579ff4-3f05-4fe4-b589-2ab741cc34d7 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '2f6e7bbfe4d74542aea3f50aa23353bf'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:07:59.131514 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-6e579ff4-3f05-4fe4-b589-2ab741cc34d7 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '2f6e7bbfe4d74542aea3f50aa23353bf'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:07:59.132264 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-6e579ff4-3f05-4fe4-b589-2ab741cc34d7 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '2f6e7bbfe4d74542aea3f50aa23353bf'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:07:59.139911 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-6e579ff4-3f05-4fe4-b589-2ab741cc34d7 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '2f6e7bbfe4d74542aea3f50aa23353bf'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:07:59.145929 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 956/1937] 10.222.0.50 () {58 vars in 1224 bytes} [Tue Sep 13 16:07:58 2022] DELETE /identity/v3/users/2f6e7bbfe4d74542aea3f50aa23353bf => generated 0 bytes in 181 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 13 16:07:59.441070 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97167383-5536-4e1d-bc84-faa92c1d7385 None None] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:07:59.455635 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97167383-5536-4e1d-bc84-faa92c1d7385 None None] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:07:59.538274 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2437fdf-53b1-4417-a5d8-5544b253abbe admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:07:59.539099 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2437fdf-53b1-4417-a5d8-5544b253abbe admin admin] REQUEST_METHOD: `DELETE` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:07:59.539469 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2437fdf-53b1-4417-a5d8-5544b253abbe admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:07:59.539815 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2437fdf-53b1-4417-a5d8-5544b253abbe admin admin] PATH_INFO: `/v3/projects/1e8eb6a8bffd4e8286ed9fd76f77dd92` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:07:59.541599 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2437fdf-53b1-4417-a5d8-5544b253abbe admin admin] RBAC: Authorizing `identity:delete_project(project_id=1e8eb6a8bffd4e8286ed9fd76f77dd92)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:07:59.543837 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2437fdf-53b1-4417-a5d8-5544b253abbe admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:07:59.602775 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-e2437fdf-53b1-4417-a5d8-5544b253abbe admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '1e8eb6a8bffd4e8286ed9fd76f77dd92', 'request_id': 'req-e2437fdf-53b1-4417-a5d8-5544b253abbe'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:07:59.611533 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-e2437fdf-53b1-4417-a5d8-5544b253abbe admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '1e8eb6a8bffd4e8286ed9fd76f77dd92', 'request_id': 'req-e2437fdf-53b1-4417-a5d8-5544b253abbe'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:07:59.620790 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-e2437fdf-53b1-4417-a5d8-5544b253abbe admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '1e8eb6a8bffd4e8286ed9fd76f77dd92', 'request_id': 'req-e2437fdf-53b1-4417-a5d8-5544b253abbe'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:07:59.634391 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-e2437fdf-53b1-4417-a5d8-5544b253abbe admin admin] The token cache is being invalidate because project 1e8eb6a8bffd4e8286ed9fd76f77dd92 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81437) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 13 16:07:59.634757 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-e2437fdf-53b1-4417-a5d8-5544b253abbe admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:07:59.638327 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-e2437fdf-53b1-4417-a5d8-5544b253abbe admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:07:59.644850 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f062c08a-7d60-4595-9447-8993a406a08d None None] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:07:59.645302 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f062c08a-7d60-4595-9447-8993a406a08d None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:07:59.645725 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f062c08a-7d60-4595-9447-8993a406a08d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:07:59.651278 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 982/1938] 10.222.0.50 () {58 vars in 1233 bytes} [Tue Sep 13 16:07:59 2022] DELETE /identity/v3/projects/1e8eb6a8bffd4e8286ed9fd76f77dd92 => generated 0 bytes in 200 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 16:07:59.676632 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.auth.core [None req-f062c08a-7d60-4595-9447-8993a406a08d None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81438) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:07:59.678005 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-f062c08a-7d60-4595-9447-8993a406a08d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:07:59.768072 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 957/1939] 10.222.0.50 () {58 vars in 958 bytes} [Tue Sep 13 16:07:59 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 124 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 13 16:07:59.776886 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c124ee67-8cf0-4c37-94f9-e64c5db7d0de None None] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:07:59.777200 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c124ee67-8cf0-4c37-94f9-e64c5db7d0de None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:07:59.777501 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c124ee67-8cf0-4c37-94f9-e64c5db7d0de None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:07:59.806711 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.auth.core [None req-c124ee67-8cf0-4c37-94f9-e64c5db7d0de None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81437) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:07:59.809277 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-c124ee67-8cf0-4c37-94f9-e64c5db7d0de None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:07:59.822064 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 983/1940] 10.222.0.50 () {58 vars in 958 bytes} [Tue Sep 13 16:07:59 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 46 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:07:59.826065 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f062c08a-7d60-4595-9447-8993a406a08d None None] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:07:59.831290 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=87,keystone:DELETE=12,keystone:INSERT=1 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:07:59.840333 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f062c08a-7d60-4595-9447-8993a406a08d None None] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:07:59.863132 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a1cede0-4cb0-4ea9-b548-e2d9183deca9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:07:59.864520 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a1cede0-4cb0-4ea9-b548-e2d9183deca9 admin admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:07:59.864986 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a1cede0-4cb0-4ea9-b548-e2d9183deca9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:07:59.865466 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a1cede0-4cb0-4ea9-b548-e2d9183deca9 admin admin] PATH_INFO: `/v3/domains` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:07:59.867755 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a1cede0-4cb0-4ea9-b548-e2d9183deca9 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81438) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 16:07:59.868658 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a1cede0-4cb0-4ea9-b548-e2d9183deca9 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:07:59.870746 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a1cede0-4cb0-4ea9-b548-e2d9183deca9 admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:07:59.895043 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 958/1941] 10.222.0.50 () {58 vars in 1153 bytes} [Tue Sep 13 16:07:59 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 69 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 13 16:07:59.900004 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c124ee67-8cf0-4c37-94f9-e64c5db7d0de None None] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:07:59.915385 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c124ee67-8cf0-4c37-94f9-e64c5db7d0de None None] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:07:59.931043 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4168d57d-c52c-400d-86b0-31818a2d546d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:07:59.931932 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4168d57d-c52c-400d-86b0-31818a2d546d admin admin] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:07:59.932235 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4168d57d-c52c-400d-86b0-31818a2d546d admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:07:59.932556 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4168d57d-c52c-400d-86b0-31818a2d546d admin admin] PATH_INFO: `/v3/projects` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:07:59.933213 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4168d57d-c52c-400d-86b0-31818a2d546d admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:07:59.935352 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4168d57d-c52c-400d-86b0-31818a2d546d admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:07:59.955417 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:07:59.983452 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 984/1942] 10.222.0.50 () {60 vars in 1153 bytes} [Tue Sep 13 16:07:59 2022] POST /identity/v3/projects => generated 385 bytes in 84 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 13 16:07:59.987273 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a1cede0-4cb0-4ea9-b548-e2d9183deca9 admin admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:07:59.999902 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a1cede0-4cb0-4ea9-b548-e2d9183deca9 admin admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:08:00.102079 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03dcaeab-8ecd-4f8e-9d9d-d04c9c91f4f9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:08:00.103089 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03dcaeab-8ecd-4f8e-9d9d-d04c9c91f4f9 admin admin] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:08:00.103504 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03dcaeab-8ecd-4f8e-9d9d-d04c9c91f4f9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:08:00.103846 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03dcaeab-8ecd-4f8e-9d9d-d04c9c91f4f9 admin admin] PATH_INFO: `/v3/users` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:08:00.104608 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03dcaeab-8ecd-4f8e-9d9d-d04c9c91f4f9 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:08:00.106603 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03dcaeab-8ecd-4f8e-9d9d-d04c9c91f4f9 admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:08:00.138928 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 959/1943] 10.222.0.50 () {60 vars in 1144 bytes} [Tue Sep 13 16:07:59 2022] POST /identity/v3/users => generated 293 bytes in 151 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 16:08:00.141832 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4168d57d-c52c-400d-86b0-31818a2d546d admin admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:08:00.152245 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4168d57d-c52c-400d-86b0-31818a2d546d admin admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:08:00.156950 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3096e275-e9b1-4927-88d9-abfb7d3d7649 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:08:00.157561 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3096e275-e9b1-4927-88d9-abfb7d3d7649 admin admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:08:00.157870 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3096e275-e9b1-4927-88d9-abfb7d3d7649 admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:08:00.158145 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3096e275-e9b1-4927-88d9-abfb7d3d7649 admin admin] PATH_INFO: `/v3/roles` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:08:00.158786 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3096e275-e9b1-4927-88d9-abfb7d3d7649 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:08:00.160428 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3096e275-e9b1-4927-88d9-abfb7d3d7649 admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:08:00.171124 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 985/1944] 10.222.0.50 () {58 vars in 1122 bytes} [Tue Sep 13 16:08:00 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 16:08:00.175093 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03dcaeab-8ecd-4f8e-9d9d-d04c9c91f4f9 admin admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:08:00.186504 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03dcaeab-8ecd-4f8e-9d9d-d04c9c91f4f9 admin admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:08:00.191513 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39d1db04-44b4-4816-9707-219e247a7056 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:08:00.192300 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39d1db04-44b4-4816-9707-219e247a7056 admin admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:08:00.192653 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39d1db04-44b4-4816-9707-219e247a7056 admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:08:00.193002 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39d1db04-44b4-4816-9707-219e247a7056 admin admin] PATH_INFO: `/v3/roles` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:08:00.193866 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39d1db04-44b4-4816-9707-219e247a7056 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:08:00.195688 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39d1db04-44b4-4816-9707-219e247a7056 admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:08:00.210945 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 960/1945] 10.222.0.50 () {58 vars in 1122 bytes} [Tue Sep 13 16:08:00 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 16:08:00.212459 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3096e275-e9b1-4927-88d9-abfb7d3d7649 admin admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:08:00.223392 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3096e275-e9b1-4927-88d9-abfb7d3d7649 admin admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:08:00.235152 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6978fe43-84ef-437a-863e-c5731b7dbbab admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:08:00.236069 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6978fe43-84ef-437a-863e-c5731b7dbbab admin admin] REQUEST_METHOD: `PUT` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:08:00.236546 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6978fe43-84ef-437a-863e-c5731b7dbbab admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:08:00.236923 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6978fe43-84ef-437a-863e-c5731b7dbbab admin admin] PATH_INFO: `/v3/projects/52eb30eacfbf4544898d44d16116e5ac/users/6b6860ce152d496397940b9baffdbd49/roles/15db4aaca3264ae29bde38903ba43471` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:08:00.253530 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6978fe43-84ef-437a-863e-c5731b7dbbab admin admin] RBAC: Authorizing `identity:create_grant(project_id=52eb30eacfbf4544898d44d16116e5ac, user_id=6b6860ce152d496397940b9baffdbd49, role_id=15db4aaca3264ae29bde38903ba43471)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:08:00.256232 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6978fe43-84ef-437a-863e-c5731b7dbbab admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:08:00.269391 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 986/1946] 10.222.0.50 () {60 vars in 1483 bytes} [Tue Sep 13 16:08:00 2022] PUT /identity/v3/projects/52eb30eacfbf4544898d44d16116e5ac/users/6b6860ce152d496397940b9baffdbd49/roles/15db4aaca3264ae29bde38903ba43471 => generated 0 bytes in 57 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 16:08:00.280720 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a62b6e07-a1ac-47b3-9a86-c0f63816fad1 None None] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:08:00.281006 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a62b6e07-a1ac-47b3-9a86-c0f63816fad1 None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:08:00.281277 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a62b6e07-a1ac-47b3-9a86-c0f63816fad1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:08:00.310618 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.auth.core [None req-a62b6e07-a1ac-47b3-9a86-c0f63816fad1 None None] MFA Rules not processed for user `6b6860ce152d496397940b9baffdbd49`. Rule list: `[]` (Enabled: `True`). {{(pid=81438) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:08:00.311907 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-a62b6e07-a1ac-47b3-9a86-c0f63816fad1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:08:00.374493 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 961/1947] 10.222.0.50 () {58 vars in 958 bytes} [Tue Sep 13 16:08:00 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:08:00.385607 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6978fe43-84ef-437a-863e-c5731b7dbbab admin admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:08:00.387335 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-6978fe43-84ef-437a-863e-c5731b7dbbab admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:08:00.447504 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6978fe43-84ef-437a-863e-c5731b7dbbab admin admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:08:00.451150 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33692496-c695-490c-b9c7-99300391a8fe service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7fd6b5afd67b48f9b0f0597ccd4cf4df', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:08:00.451760 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33692496-c695-490c-b9c7-99300391a8fe service cinder] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:08:00.452045 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33692496-c695-490c-b9c7-99300391a8fe service cinder] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:08:00.452320 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33692496-c695-490c-b9c7-99300391a8fe service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:08:00.460568 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33692496-c695-490c-b9c7-99300391a8fe service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:08:00.461991 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33692496-c695-490c-b9c7-99300391a8fe service cinder] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:08:00.470599 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 987/1948] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 16:08:00 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 85 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 16:08:10.068824 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a62b6e07-a1ac-47b3-9a86-c0f63816fad1 None None] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:08:10.153848 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:08:10.159098 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a62b6e07-a1ac-47b3-9a86-c0f63816fad1 None None] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:08:10.166729 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-858f76e3-e79a-4cea-a4ef-d6ac2c5d2ba9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:08:10.167641 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-858f76e3-e79a-4cea-a4ef-d6ac2c5d2ba9 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:08:10.168130 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-858f76e3-e79a-4cea-a4ef-d6ac2c5d2ba9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:08:10.168495 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-858f76e3-e79a-4cea-a4ef-d6ac2c5d2ba9 admin admin] PATH_INFO: `/v3/users/6b6860ce152d496397940b9baffdbd49` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:08:10.169746 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-858f76e3-e79a-4cea-a4ef-d6ac2c5d2ba9 admin admin] RBAC: Authorizing `identity:delete_user(user_id=6b6860ce152d496397940b9baffdbd49)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:08:10.172508 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-858f76e3-e79a-4cea-a4ef-d6ac2c5d2ba9 admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:08:10.253204 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33692496-c695-490c-b9c7-99300391a8fe service cinder] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:08:10.255556 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-33692496-c695-490c-b9c7-99300391a8fe service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:08:10.270855 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33692496-c695-490c-b9c7-99300391a8fe service cinder] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:08:10.280175 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-858f76e3-e79a-4cea-a4ef-d6ac2c5d2ba9 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '6b6860ce152d496397940b9baffdbd49'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:08:10.283119 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e3f1244-0656-4ded-b99c-2cf29b8fd48b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:08:10.284060 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e3f1244-0656-4ded-b99c-2cf29b8fd48b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:08:10.284424 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e3f1244-0656-4ded-b99c-2cf29b8fd48b admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:08:10.284898 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e3f1244-0656-4ded-b99c-2cf29b8fd48b admin admin] PATH_INFO: `/v3/users/5887f4924c064e76bd9a8b2d59432aa6` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:08:10.286197 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e3f1244-0656-4ded-b99c-2cf29b8fd48b admin admin] RBAC: Authorizing `identity:delete_user(user_id=5887f4924c064e76bd9a8b2d59432aa6)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:08:10.288864 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e3f1244-0656-4ded-b99c-2cf29b8fd48b admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:08:10.297536 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-858f76e3-e79a-4cea-a4ef-d6ac2c5d2ba9 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '6b6860ce152d496397940b9baffdbd49'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:08:10.328020 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-858f76e3-e79a-4cea-a4ef-d6ac2c5d2ba9 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '6b6860ce152d496397940b9baffdbd49'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:08:10.329537 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-858f76e3-e79a-4cea-a4ef-d6ac2c5d2ba9 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '6b6860ce152d496397940b9baffdbd49'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:08:10.348715 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-858f76e3-e79a-4cea-a4ef-d6ac2c5d2ba9 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '6b6860ce152d496397940b9baffdbd49'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:08:10.362831 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 962/1949] 10.222.0.50 () {58 vars in 1224 bytes} [Tue Sep 13 16:08:10 2022] DELETE /identity/v3/users/6b6860ce152d496397940b9baffdbd49 => generated 0 bytes in 292 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 16:08:10.365318 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65a357ad-335a-4a14-a28e-20303a886665 None None] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:08:10.366503 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65a357ad-335a-4a14-a28e-20303a886665 None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:08:10.367437 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65a357ad-335a-4a14-a28e-20303a886665 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:08:10.373840 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-9e3f1244-0656-4ded-b99c-2cf29b8fd48b admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '5887f4924c064e76bd9a8b2d59432aa6'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:08:10.394270 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-9e3f1244-0656-4ded-b99c-2cf29b8fd48b admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '5887f4924c064e76bd9a8b2d59432aa6'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:08:10.407535 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.auth.core [None req-65a357ad-335a-4a14-a28e-20303a886665 None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81438) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:08:10.410810 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-65a357ad-335a-4a14-a28e-20303a886665 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:08:10.418537 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-9e3f1244-0656-4ded-b99c-2cf29b8fd48b admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '5887f4924c064e76bd9a8b2d59432aa6'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:08:10.422356 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-9e3f1244-0656-4ded-b99c-2cf29b8fd48b admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '5887f4924c064e76bd9a8b2d59432aa6'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:08:10.443273 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-9e3f1244-0656-4ded-b99c-2cf29b8fd48b admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '5887f4924c064e76bd9a8b2d59432aa6'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:08:10.457507 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 988/1950] 10.222.0.50 () {58 vars in 1224 bytes} [Tue Sep 13 16:08:10 2022] DELETE /identity/v3/users/5887f4924c064e76bd9a8b2d59432aa6 => generated 0 bytes in 205 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 16:08:10.463028 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0eec66ca-93b4-4b4e-941b-f5cdc150ce53 None None] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:08:10.463298 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0eec66ca-93b4-4b4e-941b-f5cdc150ce53 None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:08:10.463719 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0eec66ca-93b4-4b4e-941b-f5cdc150ce53 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:08:10.499271 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.auth.core [None req-0eec66ca-93b4-4b4e-941b-f5cdc150ce53 None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81437) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:08:10.501010 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-0eec66ca-93b4-4b4e-941b-f5cdc150ce53 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:08:10.523006 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 963/1951] 10.222.0.50 () {58 vars in 958 bytes} [Tue Sep 13 16:08:10 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 159 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:08:10.532989 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65a357ad-335a-4a14-a28e-20303a886665 None None] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:08:10.534925 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-65a357ad-335a-4a14-a28e-20303a886665 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:08:10.535308 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=87,keystone:INSERT=3,keystone:DELETE=10 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:08:10.587911 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 989/1952] 10.222.0.50 () {58 vars in 958 bytes} [Tue Sep 13 16:08:10 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 124 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 13 16:08:10.594574 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0eec66ca-93b4-4b4e-941b-f5cdc150ce53 None None] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:08:10.596091 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-0eec66ca-93b4-4b4e-941b-f5cdc150ce53 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:08:10.614656 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65a357ad-335a-4a14-a28e-20303a886665 None None] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:08:10.657429 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:08:10.670779 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0eec66ca-93b4-4b4e-941b-f5cdc150ce53 None None] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:08:10.675472 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-debe76f1-f33c-4dae-8125-3e5b84df75e6 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f436f07939d4855a21b195bc7566244', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:08:10.676365 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-debe76f1-f33c-4dae-8125-3e5b84df75e6 service neutron] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:08:10.676664 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-debe76f1-f33c-4dae-8125-3e5b84df75e6 service neutron] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:08:10.676927 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-debe76f1-f33c-4dae-8125-3e5b84df75e6 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:08:10.686446 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02230b6e-2abd-4731-9926-7ef8666b119b service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f436f07939d4855a21b195bc7566244', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:08:10.687184 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02230b6e-2abd-4731-9926-7ef8666b119b service neutron] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:08:10.687566 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02230b6e-2abd-4731-9926-7ef8666b119b service neutron] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:08:10.687677 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-debe76f1-f33c-4dae-8125-3e5b84df75e6 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:08:10.687975 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02230b6e-2abd-4731-9926-7ef8666b119b service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:08:10.690031 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-debe76f1-f33c-4dae-8125-3e5b84df75e6 service neutron] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:08:10.698001 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02230b6e-2abd-4731-9926-7ef8666b119b service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:08:10.699548 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02230b6e-2abd-4731-9926-7ef8666b119b service neutron] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:08:10.701737 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 990/1953] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 16:08:10 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 107 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 16:08:10.711516 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 964/1954] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 16:08:10 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 179 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 13 16:08:11.427713 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-debe76f1-f33c-4dae-8125-3e5b84df75e6 service neutron] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:08:11.431706 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-debe76f1-f33c-4dae-8125-3e5b84df75e6 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:08:11.462116 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-debe76f1-f33c-4dae-8125-3e5b84df75e6 service neutron] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:08:11.472266 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b982518b-17a7-4bfc-9ded-87f0efb8e764 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:08:11.473397 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b982518b-17a7-4bfc-9ded-87f0efb8e764 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:08:11.473884 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b982518b-17a7-4bfc-9ded-87f0efb8e764 admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:08:11.474409 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b982518b-17a7-4bfc-9ded-87f0efb8e764 admin admin] PATH_INFO: `/v3/projects/52eb30eacfbf4544898d44d16116e5ac` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:08:11.476882 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b982518b-17a7-4bfc-9ded-87f0efb8e764 admin admin] RBAC: Authorizing `identity:delete_project(project_id=52eb30eacfbf4544898d44d16116e5ac)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:08:11.479113 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b982518b-17a7-4bfc-9ded-87f0efb8e764 admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:08:11.538640 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02230b6e-2abd-4731-9926-7ef8666b119b service neutron] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:08:11.540226 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-02230b6e-2abd-4731-9926-7ef8666b119b service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:08:11.563432 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02230b6e-2abd-4731-9926-7ef8666b119b service neutron] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:08:11.574142 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-803511b4-229b-4c43-a20f-d29cc1b61117 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:08:11.574975 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-803511b4-229b-4c43-a20f-d29cc1b61117 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:08:11.575087 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-803511b4-229b-4c43-a20f-d29cc1b61117 admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:08:11.575369 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-803511b4-229b-4c43-a20f-d29cc1b61117 admin admin] PATH_INFO: `/v3/projects/fe981edab10d46d5a3865fef51134564` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:08:11.576510 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-803511b4-229b-4c43-a20f-d29cc1b61117 admin admin] RBAC: Authorizing `identity:delete_project(project_id=fe981edab10d46d5a3865fef51134564)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:08:11.578425 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-803511b4-229b-4c43-a20f-d29cc1b61117 admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:08:11.633222 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-b982518b-17a7-4bfc-9ded-87f0efb8e764 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '52eb30eacfbf4544898d44d16116e5ac', 'request_id': 'req-b982518b-17a7-4bfc-9ded-87f0efb8e764'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:08:11.666011 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-b982518b-17a7-4bfc-9ded-87f0efb8e764 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '52eb30eacfbf4544898d44d16116e5ac', 'request_id': 'req-b982518b-17a7-4bfc-9ded-87f0efb8e764'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:08:11.673749 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-b982518b-17a7-4bfc-9ded-87f0efb8e764 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '52eb30eacfbf4544898d44d16116e5ac', 'request_id': 'req-b982518b-17a7-4bfc-9ded-87f0efb8e764'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:08:11.690762 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-803511b4-229b-4c43-a20f-d29cc1b61117 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'fe981edab10d46d5a3865fef51134564', 'request_id': 'req-803511b4-229b-4c43-a20f-d29cc1b61117'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:08:11.724072 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-b982518b-17a7-4bfc-9ded-87f0efb8e764 admin admin] The token cache is being invalidate because project 52eb30eacfbf4544898d44d16116e5ac was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81437) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 13 16:08:11.726057 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-b982518b-17a7-4bfc-9ded-87f0efb8e764 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:08:11.726945 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-803511b4-229b-4c43-a20f-d29cc1b61117 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'fe981edab10d46d5a3865fef51134564', 'request_id': 'req-803511b4-229b-4c43-a20f-d29cc1b61117'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:08:11.730566 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-b982518b-17a7-4bfc-9ded-87f0efb8e764 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:08:11.735176 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 991/1955] 10.222.0.50 () {58 vars in 1233 bytes} [Tue Sep 13 16:08:11 2022] DELETE /identity/v3/projects/52eb30eacfbf4544898d44d16116e5ac => generated 0 bytes in 308 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 16:08:11.741521 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8c5067c-ea8c-415c-aa00-7778114a25ea None None] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:08:11.742227 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8c5067c-ea8c-415c-aa00-7778114a25ea None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:08:11.742522 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8c5067c-ea8c-415c-aa00-7778114a25ea None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:08:11.744343 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-803511b4-229b-4c43-a20f-d29cc1b61117 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'fe981edab10d46d5a3865fef51134564', 'request_id': 'req-803511b4-229b-4c43-a20f-d29cc1b61117'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:08:11.784981 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-803511b4-229b-4c43-a20f-d29cc1b61117 admin admin] The token cache is being invalidate because project fe981edab10d46d5a3865fef51134564 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81438) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 13 16:08:11.786323 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-803511b4-229b-4c43-a20f-d29cc1b61117 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:08:11.788960 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-803511b4-229b-4c43-a20f-d29cc1b61117 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:08:11.793829 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.auth.core [None req-a8c5067c-ea8c-415c-aa00-7778114a25ea None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81437) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:08:11.796552 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-a8c5067c-ea8c-415c-aa00-7778114a25ea None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:08:11.799824 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 965/1956] 10.222.0.50 () {58 vars in 1233 bytes} [Tue Sep 13 16:08:11 2022] DELETE /identity/v3/projects/fe981edab10d46d5a3865fef51134564 => generated 0 bytes in 259 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 16:08:11.801709 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-803511b4-229b-4c43-a20f-d29cc1b61117 admin admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:08:11.807831 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-803511b4-229b-4c43-a20f-d29cc1b61117 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:08:11.948330 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:08:11.971513 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-803511b4-229b-4c43-a20f-d29cc1b61117 admin admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:08:11.979214 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 992/1957] 10.222.0.50 () {58 vars in 958 bytes} [Tue Sep 13 16:08:11 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 240 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 13 16:08:11.980856 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cff9ec83-6d18-4508-ab90-956fd5655cd8 None None] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:08:11.981332 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cff9ec83-6d18-4508-ab90-956fd5655cd8 None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:08:11.981792 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cff9ec83-6d18-4508-ab90-956fd5655cd8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:08:11.983816 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a5c2d79-7cc8-4983-b6ae-acaa163932b1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:08:11.984642 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a5c2d79-7cc8-4983-b6ae-acaa163932b1 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:08:11.985031 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a5c2d79-7cc8-4983-b6ae-acaa163932b1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:08:11.985413 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a5c2d79-7cc8-4983-b6ae-acaa163932b1 admin admin] PATH_INFO: `/v3/users/6c569801e5e34625b8ed11d688541dab` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:08:11.987558 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a5c2d79-7cc8-4983-b6ae-acaa163932b1 admin admin] RBAC: Authorizing `identity:delete_user(user_id=6c569801e5e34625b8ed11d688541dab)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:08:11.990351 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a5c2d79-7cc8-4983-b6ae-acaa163932b1 admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:08:12.049087 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.auth.core [None req-cff9ec83-6d18-4508-ab90-956fd5655cd8 None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81437) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:08:12.051551 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-cff9ec83-6d18-4508-ab90-956fd5655cd8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:08:12.075936 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 993/1958] 10.222.0.50 () {58 vars in 958 bytes} [Tue Sep 13 16:08:11 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:08:12.077948 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cff9ec83-6d18-4508-ab90-956fd5655cd8 None None] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:08:12.094569 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cff9ec83-6d18-4508-ab90-956fd5655cd8 None None] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:08:12.101203 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-5a5c2d79-7cc8-4983-b6ae-acaa163932b1 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '6c569801e5e34625b8ed11d688541dab'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:08:12.104891 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1450becd-616d-4271-b0c3-4d0e5298b370 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:08:12.108831 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1450becd-616d-4271-b0c3-4d0e5298b370 admin admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:08:12.108831 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1450becd-616d-4271-b0c3-4d0e5298b370 admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:08:12.109139 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1450becd-616d-4271-b0c3-4d0e5298b370 admin admin] PATH_INFO: `/v3/domains` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:08:12.110699 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1450becd-616d-4271-b0c3-4d0e5298b370 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81437) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 16:08:12.110699 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1450becd-616d-4271-b0c3-4d0e5298b370 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:08:12.116075 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1450becd-616d-4271-b0c3-4d0e5298b370 admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:08:12.116412 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-5a5c2d79-7cc8-4983-b6ae-acaa163932b1 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '6c569801e5e34625b8ed11d688541dab'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:08:12.127982 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=82,keystone:DELETE=16,keystone:INSERT=2 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:08:12.152074 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-5a5c2d79-7cc8-4983-b6ae-acaa163932b1 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '6c569801e5e34625b8ed11d688541dab'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:08:12.152981 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-5a5c2d79-7cc8-4983-b6ae-acaa163932b1 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '6c569801e5e34625b8ed11d688541dab'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:08:12.160223 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 994/1959] 10.222.0.50 () {58 vars in 1153 bytes} [Tue Sep 13 16:08:12 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 80 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 16:08:12.161887 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1450becd-616d-4271-b0c3-4d0e5298b370 admin admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:08:12.164053 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-1450becd-616d-4271-b0c3-4d0e5298b370 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:08:12.171070 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-5a5c2d79-7cc8-4983-b6ae-acaa163932b1 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '6c569801e5e34625b8ed11d688541dab'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:08:12.179798 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1450becd-616d-4271-b0c3-4d0e5298b370 admin admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:08:12.190308 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 966/1960] 10.222.0.50 () {58 vars in 1224 bytes} [Tue Sep 13 16:08:11 2022] DELETE /identity/v3/users/6c569801e5e34625b8ed11d688541dab => generated 0 bytes in 387 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 16:08:12.190308 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a5c2d79-7cc8-4983-b6ae-acaa163932b1 admin admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:08:12.190563 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9c8f622-267e-4d7d-9de1-2852e5585c0b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:08:12.191481 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-5a5c2d79-7cc8-4983-b6ae-acaa163932b1 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:08:12.192021 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9c8f622-267e-4d7d-9de1-2852e5585c0b admin admin] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:08:12.192367 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9c8f622-267e-4d7d-9de1-2852e5585c0b admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:08:12.192893 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9c8f622-267e-4d7d-9de1-2852e5585c0b admin admin] PATH_INFO: `/v3/projects` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:08:12.194551 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9c8f622-267e-4d7d-9de1-2852e5585c0b admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:08:12.197861 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9c8f622-267e-4d7d-9de1-2852e5585c0b admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:08:12.251946 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 995/1961] 10.222.0.50 () {60 vars in 1153 bytes} [Tue Sep 13 16:08:12 2022] POST /identity/v3/projects => generated 369 bytes in 87 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 13 16:08:12.254450 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9c8f622-267e-4d7d-9de1-2852e5585c0b admin admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:08:12.309201 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a5c2d79-7cc8-4983-b6ae-acaa163932b1 admin admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:08:12.386022 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7be2555e-1859-4693-9da1-15dac4454950 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:08:12.386437 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7be2555e-1859-4693-9da1-15dac4454950 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:08:12.386720 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7be2555e-1859-4693-9da1-15dac4454950 admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:08:12.387046 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7be2555e-1859-4693-9da1-15dac4454950 admin admin] PATH_INFO: `/v3/users/11dc0707bcf84e3784afbb0031ab90ca` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:08:12.392413 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7be2555e-1859-4693-9da1-15dac4454950 admin admin] RBAC: Authorizing `identity:delete_user(user_id=11dc0707bcf84e3784afbb0031ab90ca)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:08:12.396244 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7be2555e-1859-4693-9da1-15dac4454950 admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:08:12.406797 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9c8f622-267e-4d7d-9de1-2852e5585c0b admin admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:08:12.414428 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0375e04-fee4-4c74-b717-4603bb9ccf20 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:08:12.415110 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0375e04-fee4-4c74-b717-4603bb9ccf20 admin admin] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:08:12.415448 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0375e04-fee4-4c74-b717-4603bb9ccf20 admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:08:12.415789 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0375e04-fee4-4c74-b717-4603bb9ccf20 admin admin] PATH_INFO: `/v3/users` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:08:12.416408 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0375e04-fee4-4c74-b717-4603bb9ccf20 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:08:12.419122 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0375e04-fee4-4c74-b717-4603bb9ccf20 admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:08:12.448075 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 996/1962] 10.222.0.50 () {60 vars in 1144 bytes} [Tue Sep 13 16:08:12 2022] POST /identity/v3/users => generated 285 bytes in 195 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 13 16:08:12.452068 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0375e04-fee4-4c74-b717-4603bb9ccf20 admin admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:08:12.465906 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0375e04-fee4-4c74-b717-4603bb9ccf20 admin admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:08:12.471874 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f119a3f-5ecf-4135-840a-71729de8176b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:08:12.472509 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f119a3f-5ecf-4135-840a-71729de8176b admin admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:08:12.473164 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f119a3f-5ecf-4135-840a-71729de8176b admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:08:12.473449 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f119a3f-5ecf-4135-840a-71729de8176b admin admin] PATH_INFO: `/v3/roles` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:08:12.474386 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f119a3f-5ecf-4135-840a-71729de8176b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:08:12.476788 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f119a3f-5ecf-4135-840a-71729de8176b admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:08:12.481396 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-7be2555e-1859-4693-9da1-15dac4454950 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '11dc0707bcf84e3784afbb0031ab90ca'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:08:12.493187 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-7be2555e-1859-4693-9da1-15dac4454950 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '11dc0707bcf84e3784afbb0031ab90ca'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:08:12.499932 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 997/1963] 10.222.0.50 () {58 vars in 1122 bytes} [Tue Sep 13 16:08:12 2022] GET /identity/v3/roles => generated 1420 bytes in 48 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 16:08:12.504142 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f119a3f-5ecf-4135-840a-71729de8176b admin admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:08:12.516563 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-7be2555e-1859-4693-9da1-15dac4454950 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '11dc0707bcf84e3784afbb0031ab90ca'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:08:12.517673 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-7be2555e-1859-4693-9da1-15dac4454950 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '11dc0707bcf84e3784afbb0031ab90ca'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:08:12.517879 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f119a3f-5ecf-4135-840a-71729de8176b admin admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:08:12.525503 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26f97770-610f-4ead-85e1-b77b83c9ccdb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:08:12.526430 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26f97770-610f-4ead-85e1-b77b83c9ccdb admin admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:08:12.526964 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26f97770-610f-4ead-85e1-b77b83c9ccdb admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:08:12.527369 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26f97770-610f-4ead-85e1-b77b83c9ccdb admin admin] PATH_INFO: `/v3/roles` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:08:12.528140 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26f97770-610f-4ead-85e1-b77b83c9ccdb admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:08:12.529798 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-7be2555e-1859-4693-9da1-15dac4454950 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '11dc0707bcf84e3784afbb0031ab90ca'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:08:12.530091 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26f97770-610f-4ead-85e1-b77b83c9ccdb admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:08:12.542038 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 967/1964] 10.222.0.50 () {58 vars in 1224 bytes} [Tue Sep 13 16:08:12 2022] DELETE /identity/v3/users/11dc0707bcf84e3784afbb0031ab90ca => generated 0 bytes in 351 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 16:08:12.543941 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf8a0380-8957-400f-982d-8805ce77add9 None None] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:08:12.544234 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf8a0380-8957-400f-982d-8805ce77add9 None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:08:12.544655 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf8a0380-8957-400f-982d-8805ce77add9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:08:12.556161 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=88,keystone:DELETE=11,keystone:INSERT=1 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:08:12.573288 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 998/1965] 10.222.0.50 () {58 vars in 1122 bytes} [Tue Sep 13 16:08:12 2022] GET /identity/v3/roles => generated 1420 bytes in 70 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 16:08:12.577720 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26f97770-610f-4ead-85e1-b77b83c9ccdb admin admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:08:12.580814 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.auth.core [None req-cf8a0380-8957-400f-982d-8805ce77add9 None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81438) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:08:12.582824 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-cf8a0380-8957-400f-982d-8805ce77add9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:08:12.588018 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-26f97770-610f-4ead-85e1-b77b83c9ccdb admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:08:12.621376 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26f97770-610f-4ead-85e1-b77b83c9ccdb admin admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:08:12.665406 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 968/1966] 10.222.0.50 () {58 vars in 958 bytes} [Tue Sep 13 16:08:12 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 123 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 13 16:08:12.714016 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b10bf0f0-e58a-4491-8768-de20822e7984 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:08:12.714701 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b10bf0f0-e58a-4491-8768-de20822e7984 admin admin] REQUEST_METHOD: `PUT` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:08:12.715036 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b10bf0f0-e58a-4491-8768-de20822e7984 admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:08:12.715316 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b10bf0f0-e58a-4491-8768-de20822e7984 admin admin] PATH_INFO: `/v3/projects/718b3e0e848f4540ba0237b60e092664/users/2092fb6d24d040419b1ea48ea4945831/roles/15db4aaca3264ae29bde38903ba43471` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:08:12.736791 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b10bf0f0-e58a-4491-8768-de20822e7984 admin admin] RBAC: Authorizing `identity:create_grant(project_id=718b3e0e848f4540ba0237b60e092664, user_id=2092fb6d24d040419b1ea48ea4945831, role_id=15db4aaca3264ae29bde38903ba43471)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:08:12.740349 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b10bf0f0-e58a-4491-8768-de20822e7984 admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:08:12.740535 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf8a0380-8957-400f-982d-8805ce77add9 None None] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:08:12.742403 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-cf8a0380-8957-400f-982d-8805ce77add9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:08:12.759032 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 999/1967] 10.222.0.50 () {60 vars in 1483 bytes} [Tue Sep 13 16:08:12 2022] PUT /identity/v3/projects/718b3e0e848f4540ba0237b60e092664/users/2092fb6d24d040419b1ea48ea4945831/roles/15db4aaca3264ae29bde38903ba43471 => generated 0 bytes in 181 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 16:08:12.769182 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c61b9b5-4688-433f-be42-7cd0e30175dc None None] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:08:12.770711 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c61b9b5-4688-433f-be42-7cd0e30175dc None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:08:12.771021 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c61b9b5-4688-433f-be42-7cd0e30175dc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:08:12.808384 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:08:12.856750 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.auth.core [None req-2c61b9b5-4688-433f-be42-7cd0e30175dc None None] MFA Rules not processed for user `2092fb6d24d040419b1ea48ea4945831`. Rule list: `[]` (Enabled: `True`). {{(pid=81437) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:08:12.860366 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-2c61b9b5-4688-433f-be42-7cd0e30175dc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:08:12.866563 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf8a0380-8957-400f-982d-8805ce77add9 None None] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:08:12.970859 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1000/1968] 10.222.0.50 () {58 vars in 958 bytes} [Tue Sep 13 16:08:12 2022] POST /identity/v3/auth/tokens => generated 3246 bytes in 203 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:08:12.984222 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c61b9b5-4688-433f-be42-7cd0e30175dc None None] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:08:12.985983 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-197760a1-cf66-48da-8d80-49a3db04eded service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f436f07939d4855a21b195bc7566244', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:08:12.987204 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-2c61b9b5-4688-433f-be42-7cd0e30175dc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:08:12.987325 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-197760a1-cf66-48da-8d80-49a3db04eded service neutron] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:08:12.987598 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-197760a1-cf66-48da-8d80-49a3db04eded service neutron] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:08:12.987874 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-197760a1-cf66-48da-8d80-49a3db04eded service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:08:13.119971 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-197760a1-cf66-48da-8d80-49a3db04eded service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:08:13.121430 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-197760a1-cf66-48da-8d80-49a3db04eded service neutron] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:08:13.130272 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c61b9b5-4688-433f-be42-7cd0e30175dc None None] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:08:13.134010 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 969/1969] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 16:08:12 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 394 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 13 16:08:13.140327 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b6926ac-2a00-41ca-9c8c-95a92021ec26 service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6ab5f8018f464be69d4af7683f553738', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:08:13.141416 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b6926ac-2a00-41ca-9c8c-95a92021ec26 service glance] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:08:13.141696 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b6926ac-2a00-41ca-9c8c-95a92021ec26 service glance] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:08:13.141972 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b6926ac-2a00-41ca-9c8c-95a92021ec26 service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:08:13.151733 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b6926ac-2a00-41ca-9c8c-95a92021ec26 service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:08:13.153331 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b6926ac-2a00-41ca-9c8c-95a92021ec26 service glance] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:08:13.164328 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1001/1970] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 16:08:12 2022] GET /identity/v3/auth/tokens => generated 3246 bytes in 180 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 13 16:08:13.986025 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-197760a1-cf66-48da-8d80-49a3db04eded service neutron] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:08:13.987628 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-197760a1-cf66-48da-8d80-49a3db04eded service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:08:14.018255 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-197760a1-cf66-48da-8d80-49a3db04eded service neutron] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:08:14.034091 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2856f09d-fbd6-496c-9a7c-208ee77e9764 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:08:14.052517 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2856f09d-fbd6-496c-9a7c-208ee77e9764 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:08:14.054071 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2856f09d-fbd6-496c-9a7c-208ee77e9764 admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:08:14.055116 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2856f09d-fbd6-496c-9a7c-208ee77e9764 admin admin] PATH_INFO: `/v3/projects/06471f253f8040628a6bb747e757c7e5` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:08:14.067300 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2856f09d-fbd6-496c-9a7c-208ee77e9764 admin admin] RBAC: Authorizing `identity:delete_project(project_id=06471f253f8040628a6bb747e757c7e5)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:08:14.077636 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2856f09d-fbd6-496c-9a7c-208ee77e9764 admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:08:14.108658 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:08:14.192006 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-2856f09d-fbd6-496c-9a7c-208ee77e9764 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '06471f253f8040628a6bb747e757c7e5', 'request_id': 'req-2856f09d-fbd6-496c-9a7c-208ee77e9764'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:08:14.212527 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-2856f09d-fbd6-496c-9a7c-208ee77e9764 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '06471f253f8040628a6bb747e757c7e5', 'request_id': 'req-2856f09d-fbd6-496c-9a7c-208ee77e9764'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:08:14.222147 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-2856f09d-fbd6-496c-9a7c-208ee77e9764 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '06471f253f8040628a6bb747e757c7e5', 'request_id': 'req-2856f09d-fbd6-496c-9a7c-208ee77e9764'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:08:14.263019 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-2856f09d-fbd6-496c-9a7c-208ee77e9764 admin admin] The token cache is being invalidate because project 06471f253f8040628a6bb747e757c7e5 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81438) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 13 16:08:14.263539 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-2856f09d-fbd6-496c-9a7c-208ee77e9764 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:08:14.264762 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-2856f09d-fbd6-496c-9a7c-208ee77e9764 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:08:14.268125 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 970/1971] 10.222.0.50 () {58 vars in 1233 bytes} [Tue Sep 13 16:08:13 2022] DELETE /identity/v3/projects/06471f253f8040628a6bb747e757c7e5 => generated 0 bytes in 283 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 16:08:14.273618 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-649a5ec7-2f10-4648-9a4c-cad74d9accd0 None None] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:08:14.274252 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-649a5ec7-2f10-4648-9a4c-cad74d9accd0 None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:08:14.274359 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-649a5ec7-2f10-4648-9a4c-cad74d9accd0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:08:14.329471 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.auth.core [None req-649a5ec7-2f10-4648-9a4c-cad74d9accd0 None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81437) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:08:14.330960 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-649a5ec7-2f10-4648-9a4c-cad74d9accd0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:08:14.365470 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2856f09d-fbd6-496c-9a7c-208ee77e9764 admin admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:08:14.367284 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-2856f09d-fbd6-496c-9a7c-208ee77e9764 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:08:14.468122 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1002/1972] 10.222.0.50 () {58 vars in 958 bytes} [Tue Sep 13 16:08:14 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 195 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 13 16:08:14.482553 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d7a0819-5115-45ea-a25c-a1fdf253d899 None None] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:08:14.482841 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d7a0819-5115-45ea-a25c-a1fdf253d899 None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:08:14.483172 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d7a0819-5115-45ea-a25c-a1fdf253d899 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:08:14.520537 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.auth.core [None req-8d7a0819-5115-45ea-a25c-a1fdf253d899 None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81437) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:08:14.523975 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2856f09d-fbd6-496c-9a7c-208ee77e9764 admin admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:08:14.525313 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-8d7a0819-5115-45ea-a25c-a1fdf253d899 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:08:14.543006 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc2331f6-008e-4454-a904-8ca18b903b5f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:08:14.546673 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc2331f6-008e-4454-a904-8ca18b903b5f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:08:14.547027 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc2331f6-008e-4454-a904-8ca18b903b5f admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:08:14.547275 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc2331f6-008e-4454-a904-8ca18b903b5f admin admin] PATH_INFO: `/v3/projects/87c0bc925183409993d04ed92503420f` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:08:14.552280 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc2331f6-008e-4454-a904-8ca18b903b5f admin admin] RBAC: Authorizing `identity:delete_project(project_id=87c0bc925183409993d04ed92503420f)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:08:14.559064 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1003/1973] 10.222.0.50 () {58 vars in 958 bytes} [Tue Sep 13 16:08:14 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 78 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:08:14.561805 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d7a0819-5115-45ea-a25c-a1fdf253d899 None None] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:08:14.564345 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc2331f6-008e-4454-a904-8ca18b903b5f admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:08:14.608423 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d7a0819-5115-45ea-a25c-a1fdf253d899 None None] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:08:14.621112 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40aff324-4e7a-45f6-84d1-6219e227b7a9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:08:14.622011 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40aff324-4e7a-45f6-84d1-6219e227b7a9 admin admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:08:14.624177 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40aff324-4e7a-45f6-84d1-6219e227b7a9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:08:14.624719 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40aff324-4e7a-45f6-84d1-6219e227b7a9 admin admin] PATH_INFO: `/v3/domains` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:08:14.625604 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40aff324-4e7a-45f6-84d1-6219e227b7a9 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81437) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 16:08:14.627094 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40aff324-4e7a-45f6-84d1-6219e227b7a9 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:08:14.634369 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40aff324-4e7a-45f6-84d1-6219e227b7a9 admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:08:14.659161 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:08:14.669030 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1004/1974] 10.222.0.50 () {58 vars in 1153 bytes} [Tue Sep 13 16:08:14 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 109 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 16:08:14.672437 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40aff324-4e7a-45f6-84d1-6219e227b7a9 admin admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:08:14.695031 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40aff324-4e7a-45f6-84d1-6219e227b7a9 admin admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:08:14.751507 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-dc2331f6-008e-4454-a904-8ca18b903b5f admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '87c0bc925183409993d04ed92503420f', 'request_id': 'req-dc2331f6-008e-4454-a904-8ca18b903b5f'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:08:14.766436 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-dc2331f6-008e-4454-a904-8ca18b903b5f admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '87c0bc925183409993d04ed92503420f', 'request_id': 'req-dc2331f6-008e-4454-a904-8ca18b903b5f'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:08:14.786165 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-dc2331f6-008e-4454-a904-8ca18b903b5f admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '87c0bc925183409993d04ed92503420f', 'request_id': 'req-dc2331f6-008e-4454-a904-8ca18b903b5f'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:08:14.835540 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-dc2331f6-008e-4454-a904-8ca18b903b5f admin admin] The token cache is being invalidate because project 87c0bc925183409993d04ed92503420f was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81438) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 13 16:08:14.835706 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-dc2331f6-008e-4454-a904-8ca18b903b5f admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:08:14.837481 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-dc2331f6-008e-4454-a904-8ca18b903b5f admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:08:14.840710 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 971/1975] 10.222.0.50 () {58 vars in 1233 bytes} [Tue Sep 13 16:08:14 2022] DELETE /identity/v3/projects/87c0bc925183409993d04ed92503420f => generated 0 bytes in 476 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 13 16:08:14.845923 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc2331f6-008e-4454-a904-8ca18b903b5f admin admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:08:14.849465 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-dc2331f6-008e-4454-a904-8ca18b903b5f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:08:14.864242 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-371288d6-ff1f-4141-9760-df3c4c16deb6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:08:14.864901 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-371288d6-ff1f-4141-9760-df3c4c16deb6 admin admin] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:08:14.865186 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-371288d6-ff1f-4141-9760-df3c4c16deb6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:08:14.866100 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-371288d6-ff1f-4141-9760-df3c4c16deb6 admin admin] PATH_INFO: `/v3/projects` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:08:14.866912 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-371288d6-ff1f-4141-9760-df3c4c16deb6 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:08:14.871965 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-371288d6-ff1f-4141-9760-df3c4c16deb6 admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:08:14.887058 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc2331f6-008e-4454-a904-8ca18b903b5f admin admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:08:14.921910 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fec7fa8f-8eb0-480f-84c4-7d213aac2ed3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:08:14.924639 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fec7fa8f-8eb0-480f-84c4-7d213aac2ed3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:08:14.925035 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fec7fa8f-8eb0-480f-84c4-7d213aac2ed3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:08:14.926014 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fec7fa8f-8eb0-480f-84c4-7d213aac2ed3 admin admin] PATH_INFO: `/v3/users/680f53a2509242c8b5143a8c6070c275` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:08:14.929348 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fec7fa8f-8eb0-480f-84c4-7d213aac2ed3 admin admin] RBAC: Authorizing `identity:delete_user(user_id=680f53a2509242c8b5143a8c6070c275)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:08:14.936688 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fec7fa8f-8eb0-480f-84c4-7d213aac2ed3 admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:08:14.938450 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1005/1976] 10.222.0.50 () {60 vars in 1153 bytes} [Tue Sep 13 16:08:14 2022] POST /identity/v3/projects => generated 367 bytes in 266 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 16:08:14.942506 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-371288d6-ff1f-4141-9760-df3c4c16deb6 admin admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:08:14.947095 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-371288d6-ff1f-4141-9760-df3c4c16deb6 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:08:14.971036 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=82,keystone:DELETE=16,keystone:INSERT=2 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:08:15.063477 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-fec7fa8f-8eb0-480f-84c4-7d213aac2ed3 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '680f53a2509242c8b5143a8c6070c275'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:08:15.094868 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-fec7fa8f-8eb0-480f-84c4-7d213aac2ed3 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '680f53a2509242c8b5143a8c6070c275'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:08:15.112080 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-fec7fa8f-8eb0-480f-84c4-7d213aac2ed3 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '680f53a2509242c8b5143a8c6070c275'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:08:15.113113 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-fec7fa8f-8eb0-480f-84c4-7d213aac2ed3 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '680f53a2509242c8b5143a8c6070c275'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:08:15.123799 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-fec7fa8f-8eb0-480f-84c4-7d213aac2ed3 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '680f53a2509242c8b5143a8c6070c275'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:08:15.135599 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 972/1977] 10.222.0.50 () {58 vars in 1224 bytes} [Tue Sep 13 16:08:14 2022] DELETE /identity/v3/users/680f53a2509242c8b5143a8c6070c275 => generated 0 bytes in 287 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 16:08:15.139168 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-371288d6-ff1f-4141-9760-df3c4c16deb6 admin admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:08:15.240115 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a210441-1e79-4ff4-bcbf-ae33edb0cbb0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:08:15.241380 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a210441-1e79-4ff4-bcbf-ae33edb0cbb0 admin admin] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:08:15.241667 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a210441-1e79-4ff4-bcbf-ae33edb0cbb0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:08:15.241941 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a210441-1e79-4ff4-bcbf-ae33edb0cbb0 admin admin] PATH_INFO: `/v3/users` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:08:15.242589 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a210441-1e79-4ff4-bcbf-ae33edb0cbb0 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:08:15.244897 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a210441-1e79-4ff4-bcbf-ae33edb0cbb0 admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:08:15.269913 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1006/1978] 10.222.0.50 () {60 vars in 1144 bytes} [Tue Sep 13 16:08:14 2022] POST /identity/v3/users => generated 284 bytes in 328 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 13 16:08:15.274112 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fec7fa8f-8eb0-480f-84c4-7d213aac2ed3 admin admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:08:15.275971 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-fec7fa8f-8eb0-480f-84c4-7d213aac2ed3 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:08:15.297425 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fec7fa8f-8eb0-480f-84c4-7d213aac2ed3 admin admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:08:15.323026 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-930a9f22-9fa1-46ad-88f6-43b0b0e3da09 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:08:15.324560 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-930a9f22-9fa1-46ad-88f6-43b0b0e3da09 admin admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:08:15.324560 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-930a9f22-9fa1-46ad-88f6-43b0b0e3da09 admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:08:15.324560 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-930a9f22-9fa1-46ad-88f6-43b0b0e3da09 admin admin] PATH_INFO: `/v3/roles` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:08:15.325404 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-930a9f22-9fa1-46ad-88f6-43b0b0e3da09 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:08:15.327580 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-930a9f22-9fa1-46ad-88f6-43b0b0e3da09 admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:08:15.364227 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 973/1979] 10.222.0.50 () {58 vars in 1122 bytes} [Tue Sep 13 16:08:15 2022] GET /identity/v3/roles => generated 1420 bytes in 90 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 13 16:08:15.369457 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a210441-1e79-4ff4-bcbf-ae33edb0cbb0 admin admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:08:15.384740 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a210441-1e79-4ff4-bcbf-ae33edb0cbb0 admin admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:08:15.408178 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2334c1eb-bf2f-4d3f-b778-de59e73631d6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:08:15.409031 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2334c1eb-bf2f-4d3f-b778-de59e73631d6 admin admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:08:15.409810 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2334c1eb-bf2f-4d3f-b778-de59e73631d6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:08:15.410206 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2334c1eb-bf2f-4d3f-b778-de59e73631d6 admin admin] PATH_INFO: `/v3/roles` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:08:15.411082 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2334c1eb-bf2f-4d3f-b778-de59e73631d6 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:08:15.413398 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2334c1eb-bf2f-4d3f-b778-de59e73631d6 admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:08:15.434949 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1007/1980] 10.222.0.50 () {58 vars in 1122 bytes} [Tue Sep 13 16:08:15 2022] GET /identity/v3/roles => generated 1420 bytes in 67 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 16:08:15.440575 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-930a9f22-9fa1-46ad-88f6-43b0b0e3da09 admin admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:08:15.453846 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-930a9f22-9fa1-46ad-88f6-43b0b0e3da09 admin admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:08:15.461706 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7aa13905-e78a-4146-a564-46722cd9054f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:08:15.462677 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7aa13905-e78a-4146-a564-46722cd9054f admin admin] REQUEST_METHOD: `PUT` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:08:15.463109 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7aa13905-e78a-4146-a564-46722cd9054f admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:08:15.463845 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7aa13905-e78a-4146-a564-46722cd9054f admin admin] PATH_INFO: `/v3/projects/e9d2efd0f85c48ae8dd78b3a68daedff/users/f7997f715f564aa0b32c5f9d383264a6/roles/15db4aaca3264ae29bde38903ba43471` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:08:15.511402 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7aa13905-e78a-4146-a564-46722cd9054f admin admin] RBAC: Authorizing `identity:create_grant(project_id=e9d2efd0f85c48ae8dd78b3a68daedff, user_id=f7997f715f564aa0b32c5f9d383264a6, role_id=15db4aaca3264ae29bde38903ba43471)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:08:15.516739 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7aa13905-e78a-4146-a564-46722cd9054f admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:08:15.553755 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 974/1981] 10.222.0.50 () {60 vars in 1483 bytes} [Tue Sep 13 16:08:15 2022] PUT /identity/v3/projects/e9d2efd0f85c48ae8dd78b3a68daedff/users/f7997f715f564aa0b32c5f9d383264a6/roles/15db4aaca3264ae29bde38903ba43471 => generated 0 bytes in 104 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 16:08:15.559257 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13dcabf4-6079-43c8-9fdc-c940808f39e3 None None] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:08:15.561378 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13dcabf4-6079-43c8-9fdc-c940808f39e3 None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:08:15.561723 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13dcabf4-6079-43c8-9fdc-c940808f39e3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:08:15.590927 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:08:15.602806 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.auth.core [None req-13dcabf4-6079-43c8-9fdc-c940808f39e3 None None] MFA Rules not processed for user `f7997f715f564aa0b32c5f9d383264a6`. Rule list: `[]` (Enabled: `True`). {{(pid=81437) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:08:15.606129 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-13dcabf4-6079-43c8-9fdc-c940808f39e3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:08:15.751467 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7aa13905-e78a-4146-a564-46722cd9054f admin admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:08:15.754509 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1008/1982] 10.222.0.50 () {58 vars in 958 bytes} [Tue Sep 13 16:08:15 2022] POST /identity/v3/auth/tokens => generated 3244 bytes in 197 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:08:15.849166 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7aa13905-e78a-4146-a564-46722cd9054f admin admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:08:15.858179 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7165f66e-3ca3-4d76-adf7-3c33e63e331a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:08:15.859169 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7165f66e-3ca3-4d76-adf7-3c33e63e331a admin admin] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:08:15.859709 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7165f66e-3ca3-4d76-adf7-3c33e63e331a admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:08:15.860201 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7165f66e-3ca3-4d76-adf7-3c33e63e331a admin admin] PATH_INFO: `/v3/projects` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:08:15.861067 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7165f66e-3ca3-4d76-adf7-3c33e63e331a admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:08:15.863311 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7165f66e-3ca3-4d76-adf7-3c33e63e331a admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:08:15.914712 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 975/1983] 10.222.0.50 () {60 vars in 1153 bytes} [Tue Sep 13 16:08:15 2022] POST /identity/v3/projects => generated 367 bytes in 161 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 16:08:15.916735 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13dcabf4-6079-43c8-9fdc-c940808f39e3 None None] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:08:15.931696 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13dcabf4-6079-43c8-9fdc-c940808f39e3 None None] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:08:15.999408 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7165f66e-3ca3-4d76-adf7-3c33e63e331a admin admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:08:16.001325 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-7165f66e-3ca3-4d76-adf7-3c33e63e331a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:08:16.034726 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b83f027-b177-45fd-b3ab-75bfb5dc8647 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:08:16.035410 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b83f027-b177-45fd-b3ab-75bfb5dc8647 admin admin] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:08:16.043024 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b83f027-b177-45fd-b3ab-75bfb5dc8647 admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:08:16.043369 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b83f027-b177-45fd-b3ab-75bfb5dc8647 admin admin] PATH_INFO: `/v3/users` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:08:16.044496 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b83f027-b177-45fd-b3ab-75bfb5dc8647 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:08:16.046822 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b83f027-b177-45fd-b3ab-75bfb5dc8647 admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:08:16.094220 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1009/1984] 10.222.0.50 () {60 vars in 1144 bytes} [Tue Sep 13 16:08:15 2022] POST /identity/v3/users => generated 290 bytes in 179 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 13 16:08:16.098243 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b83f027-b177-45fd-b3ab-75bfb5dc8647 admin admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:08:16.123935 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b83f027-b177-45fd-b3ab-75bfb5dc8647 admin admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:08:16.136490 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f53cd5ad-f2b6-4d20-86b6-4f949be10ac8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:08:16.137674 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f53cd5ad-f2b6-4d20-86b6-4f949be10ac8 admin admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:08:16.141414 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f53cd5ad-f2b6-4d20-86b6-4f949be10ac8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:08:16.141714 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f53cd5ad-f2b6-4d20-86b6-4f949be10ac8 admin admin] PATH_INFO: `/v3/roles` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:08:16.142817 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f53cd5ad-f2b6-4d20-86b6-4f949be10ac8 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:08:16.146059 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f53cd5ad-f2b6-4d20-86b6-4f949be10ac8 admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:08:16.173572 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1010/1985] 10.222.0.50 () {58 vars in 1122 bytes} [Tue Sep 13 16:08:16 2022] GET /identity/v3/roles => generated 1420 bytes in 76 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 16:08:16.177256 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f53cd5ad-f2b6-4d20-86b6-4f949be10ac8 admin admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:08:16.180077 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:DELETE=8,keystone:SELECT=89,keystone:INSERT=3 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:08:16.209696 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f53cd5ad-f2b6-4d20-86b6-4f949be10ac8 admin admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:08:16.215309 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d20622cc-8067-4e44-9289-0045c3d1da60 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:08:16.216176 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d20622cc-8067-4e44-9289-0045c3d1da60 admin admin] REQUEST_METHOD: `PUT` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:08:16.216899 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d20622cc-8067-4e44-9289-0045c3d1da60 admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:08:16.217332 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d20622cc-8067-4e44-9289-0045c3d1da60 admin admin] PATH_INFO: `/v3/projects/b19cf4202269486cae76f6df5b4a8635/users/6036198a0936436bbb563def1464dc04/roles/031d4e1a23e34f71858fc4a6dfbd4ccb` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:08:16.219368 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7165f66e-3ca3-4d76-adf7-3c33e63e331a admin admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:08:16.230291 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0ac130d-f6d1-40cd-a32d-fc2ebf232e6a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:08:16.230873 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0ac130d-f6d1-40cd-a32d-fc2ebf232e6a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:08:16.232315 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0ac130d-f6d1-40cd-a32d-fc2ebf232e6a admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:08:16.232620 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0ac130d-f6d1-40cd-a32d-fc2ebf232e6a admin admin] PATH_INFO: `/v3/projects/cef6de435b554306a7e68ac6918f8244` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:08:16.234370 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0ac130d-f6d1-40cd-a32d-fc2ebf232e6a admin admin] RBAC: Authorizing `identity:delete_project(project_id=cef6de435b554306a7e68ac6918f8244)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:08:16.237105 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0ac130d-f6d1-40cd-a32d-fc2ebf232e6a admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:08:16.264641 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d20622cc-8067-4e44-9289-0045c3d1da60 admin admin] RBAC: Authorizing `identity:create_grant(project_id=b19cf4202269486cae76f6df5b4a8635, user_id=6036198a0936436bbb563def1464dc04, role_id=031d4e1a23e34f71858fc4a6dfbd4ccb)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:08:16.272298 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d20622cc-8067-4e44-9289-0045c3d1da60 admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:08:16.302084 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1011/1986] 10.222.0.50 () {60 vars in 1483 bytes} [Tue Sep 13 16:08:16 2022] PUT /identity/v3/projects/b19cf4202269486cae76f6df5b4a8635/users/6036198a0936436bbb563def1464dc04/roles/031d4e1a23e34f71858fc4a6dfbd4ccb => generated 0 bytes in 124 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 16:08:16.305562 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d20622cc-8067-4e44-9289-0045c3d1da60 admin admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:08:16.316027 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-e0ac130d-f6d1-40cd-a32d-fc2ebf232e6a admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'cef6de435b554306a7e68ac6918f8244', 'request_id': 'req-e0ac130d-f6d1-40cd-a32d-fc2ebf232e6a'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:08:16.329079 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-e0ac130d-f6d1-40cd-a32d-fc2ebf232e6a admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'cef6de435b554306a7e68ac6918f8244', 'request_id': 'req-e0ac130d-f6d1-40cd-a32d-fc2ebf232e6a'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:08:16.342213 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-e0ac130d-f6d1-40cd-a32d-fc2ebf232e6a admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'cef6de435b554306a7e68ac6918f8244', 'request_id': 'req-e0ac130d-f6d1-40cd-a32d-fc2ebf232e6a'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:08:16.359236 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-e0ac130d-f6d1-40cd-a32d-fc2ebf232e6a admin admin] The token cache is being invalidate because project cef6de435b554306a7e68ac6918f8244 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81438) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 13 16:08:16.364412 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-e0ac130d-f6d1-40cd-a32d-fc2ebf232e6a admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:08:16.366287 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-e0ac130d-f6d1-40cd-a32d-fc2ebf232e6a admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:08:16.368685 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 976/1987] 10.222.0.50 () {58 vars in 1233 bytes} [Tue Sep 13 16:08:15 2022] DELETE /identity/v3/projects/cef6de435b554306a7e68ac6918f8244 => generated 0 bytes in 370 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 16:08:16.373584 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1027e44d-65ae-4496-b817-d0a229890845 None None] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:08:16.374180 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1027e44d-65ae-4496-b817-d0a229890845 None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:08:16.374601 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1027e44d-65ae-4496-b817-d0a229890845 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:08:16.407457 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.auth.core [None req-1027e44d-65ae-4496-b817-d0a229890845 None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81438) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:08:16.409426 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-1027e44d-65ae-4496-b817-d0a229890845 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:08:16.437827 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:08:16.445531 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d20622cc-8067-4e44-9289-0045c3d1da60 admin admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:08:16.510770 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 977/1988] 10.222.0.50 () {58 vars in 958 bytes} [Tue Sep 13 16:08:16 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 138 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:08:16.512667 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0184553d-5cb7-44d5-a0ef-98d3097e2250 None None] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:08:16.512964 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0184553d-5cb7-44d5-a0ef-98d3097e2250 None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:08:16.513248 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0184553d-5cb7-44d5-a0ef-98d3097e2250 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:08:16.533800 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4e1f941-b300-438c-b9cb-2038d0ecebdc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:08:16.546925 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4e1f941-b300-438c-b9cb-2038d0ecebdc admin admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:08:16.547249 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4e1f941-b300-438c-b9cb-2038d0ecebdc admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:08:16.547508 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4e1f941-b300-438c-b9cb-2038d0ecebdc admin admin] PATH_INFO: `/v3/roles` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:08:16.548735 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4e1f941-b300-438c-b9cb-2038d0ecebdc admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:08:16.550529 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4e1f941-b300-438c-b9cb-2038d0ecebdc admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:08:16.555738 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.auth.core [None req-0184553d-5cb7-44d5-a0ef-98d3097e2250 None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81438) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:08:16.556860 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-0184553d-5cb7-44d5-a0ef-98d3097e2250 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:08:16.567657 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1012/1989] 10.222.0.50 () {58 vars in 1122 bytes} [Tue Sep 13 16:08:16 2022] GET /identity/v3/roles => generated 1420 bytes in 263 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 16:08:16.568687 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4e1f941-b300-438c-b9cb-2038d0ecebdc admin admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:08:16.570530 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-b4e1f941-b300-438c-b9cb-2038d0ecebdc admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:08:16.572401 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 978/1990] 10.222.0.50 () {58 vars in 958 bytes} [Tue Sep 13 16:08:16 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 61 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:08:16.577158 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0184553d-5cb7-44d5-a0ef-98d3097e2250 None None] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:08:16.591563 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4e1f941-b300-438c-b9cb-2038d0ecebdc admin admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:08:16.594003 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0184553d-5cb7-44d5-a0ef-98d3097e2250 None None] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:08:16.606177 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b65c2062-d06a-4120-83e7-bb961a8f889c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:08:16.607601 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b65c2062-d06a-4120-83e7-bb961a8f889c admin admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:08:16.608088 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b65c2062-d06a-4120-83e7-bb961a8f889c admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:08:16.608209 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a4c1a55-8f0b-469b-b594-68fc794c2a24 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:08:16.608666 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b65c2062-d06a-4120-83e7-bb961a8f889c admin admin] PATH_INFO: `/v3/domains` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:08:16.608817 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a4c1a55-8f0b-469b-b594-68fc794c2a24 admin admin] REQUEST_METHOD: `PUT` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:08:16.609101 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a4c1a55-8f0b-469b-b594-68fc794c2a24 admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:08:16.609457 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a4c1a55-8f0b-469b-b594-68fc794c2a24 admin admin] PATH_INFO: `/v3/projects/b19cf4202269486cae76f6df5b4a8635/users/6036198a0936436bbb563def1464dc04/roles/031d4e1a23e34f71858fc4a6dfbd4ccb` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:08:16.610279 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b65c2062-d06a-4120-83e7-bb961a8f889c admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81438) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 16:08:16.610590 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b65c2062-d06a-4120-83e7-bb961a8f889c admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:08:16.612470 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b65c2062-d06a-4120-83e7-bb961a8f889c admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:08:16.612690 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a4c1a55-8f0b-469b-b594-68fc794c2a24 admin admin] RBAC: Authorizing `identity:create_grant(project_id=b19cf4202269486cae76f6df5b4a8635, user_id=6036198a0936436bbb563def1464dc04, role_id=031d4e1a23e34f71858fc4a6dfbd4ccb)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:08:16.616434 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a4c1a55-8f0b-469b-b594-68fc794c2a24 admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:08:16.629864 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 979/1991] 10.222.0.50 () {58 vars in 1153 bytes} [Tue Sep 13 16:08:16 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 54 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 16:08:16.632002 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1013/1992] 10.222.0.50 () {60 vars in 1483 bytes} [Tue Sep 13 16:08:16 2022] PUT /identity/v3/projects/b19cf4202269486cae76f6df5b4a8635/users/6036198a0936436bbb563def1464dc04/roles/031d4e1a23e34f71858fc4a6dfbd4ccb => generated 0 bytes in 63 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 16:08:16.635045 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b65c2062-d06a-4120-83e7-bb961a8f889c admin admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:08:16.645130 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70644f45-7d14-469e-87fd-59b15cf68fbb None None] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:08:16.645732 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70644f45-7d14-469e-87fd-59b15cf68fbb None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:08:16.646176 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70644f45-7d14-469e-87fd-59b15cf68fbb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:08:16.653267 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b65c2062-d06a-4120-83e7-bb961a8f889c admin admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:08:16.690367 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.auth.core [None req-70644f45-7d14-469e-87fd-59b15cf68fbb None None] MFA Rules not processed for user `6036198a0936436bbb563def1464dc04`. Rule list: `[]` (Enabled: `True`). {{(pid=81437) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:08:16.691789 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-70644f45-7d14-469e-87fd-59b15cf68fbb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:08:16.720642 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:08:16.729039 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13448f7e-0c5f-4bea-88e2-1faa91745895 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:08:16.730059 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13448f7e-0c5f-4bea-88e2-1faa91745895 admin admin] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:08:16.730059 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13448f7e-0c5f-4bea-88e2-1faa91745895 admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:08:16.730262 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13448f7e-0c5f-4bea-88e2-1faa91745895 admin admin] PATH_INFO: `/v3/projects` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:08:16.731026 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13448f7e-0c5f-4bea-88e2-1faa91745895 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:08:16.733177 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13448f7e-0c5f-4bea-88e2-1faa91745895 admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:08:16.776266 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 980/1993] 10.222.0.50 () {60 vars in 1153 bytes} [Tue Sep 13 16:08:16 2022] POST /identity/v3/projects => generated 365 bytes in 141 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 16:08:16.783019 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13448f7e-0c5f-4bea-88e2-1faa91745895 admin admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:08:16.791365 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1014/1994] 10.222.0.50 () {58 vars in 958 bytes} [Tue Sep 13 16:08:16 2022] POST /identity/v3/auth/tokens => generated 3311 bytes in 143 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:08:16.799205 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70644f45-7d14-469e-87fd-59b15cf68fbb None None] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:08:16.801561 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-70644f45-7d14-469e-87fd-59b15cf68fbb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:08:16.873707 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70644f45-7d14-469e-87fd-59b15cf68fbb None None] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:08:16.874056 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13448f7e-0c5f-4bea-88e2-1faa91745895 admin admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:08:16.879533 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a1a7c93-8b5a-4bf0-b5da-0ca2275f1ea8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:08:16.880602 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a1a7c93-8b5a-4bf0-b5da-0ca2275f1ea8 admin admin] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:08:16.880602 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a1a7c93-8b5a-4bf0-b5da-0ca2275f1ea8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:08:16.881035 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a1a7c93-8b5a-4bf0-b5da-0ca2275f1ea8 admin admin] PATH_INFO: `/v3/users` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:08:16.882207 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a1a7c93-8b5a-4bf0-b5da-0ca2275f1ea8 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:08:16.884256 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a1a7c93-8b5a-4bf0-b5da-0ca2275f1ea8 admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:08:16.909587 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:08:16.916941 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 981/1995] 10.222.0.50 () {60 vars in 1144 bytes} [Tue Sep 13 16:08:16 2022] POST /identity/v3/users => generated 283 bytes in 134 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 16:08:16.921149 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a1a7c93-8b5a-4bf0-b5da-0ca2275f1ea8 admin admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:08:16.933837 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a1a7c93-8b5a-4bf0-b5da-0ca2275f1ea8 admin admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:08:16.939448 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41e33e0b-d0fb-44b4-9431-fab937035468 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:08:16.940059 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41e33e0b-d0fb-44b4-9431-fab937035468 admin admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:08:16.940341 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41e33e0b-d0fb-44b4-9431-fab937035468 admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:08:16.940629 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41e33e0b-d0fb-44b4-9431-fab937035468 admin admin] PATH_INFO: `/v3/roles` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:08:16.941247 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41e33e0b-d0fb-44b4-9431-fab937035468 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:08:16.942829 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41e33e0b-d0fb-44b4-9431-fab937035468 admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:08:16.945908 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cdebbcc-4c1c-4c06-a82d-c074be668e3e service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7fd6b5afd67b48f9b0f0597ccd4cf4df', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:08:16.946579 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cdebbcc-4c1c-4c06-a82d-c074be668e3e service cinder] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:08:16.946888 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cdebbcc-4c1c-4c06-a82d-c074be668e3e service cinder] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:08:16.947202 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cdebbcc-4c1c-4c06-a82d-c074be668e3e service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:08:16.953905 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 982/1996] 10.222.0.50 () {58 vars in 1122 bytes} [Tue Sep 13 16:08:16 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 13 16:08:16.957843 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41e33e0b-d0fb-44b4-9431-fab937035468 admin admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:08:16.971308 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41e33e0b-d0fb-44b4-9431-fab937035468 admin admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:08:16.976974 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bf8c1be-77c0-4085-a418-4199f43b22ea admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:08:16.977585 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bf8c1be-77c0-4085-a418-4199f43b22ea admin admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:08:16.977862 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bf8c1be-77c0-4085-a418-4199f43b22ea admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:08:16.978132 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bf8c1be-77c0-4085-a418-4199f43b22ea admin admin] PATH_INFO: `/v3/roles` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:08:16.978770 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bf8c1be-77c0-4085-a418-4199f43b22ea admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:08:16.980403 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bf8c1be-77c0-4085-a418-4199f43b22ea admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:08:17.005755 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 983/1997] 10.222.0.50 () {58 vars in 1122 bytes} [Tue Sep 13 16:08:16 2022] GET /identity/v3/roles => generated 1420 bytes in 48 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 13 16:08:17.007929 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bf8c1be-77c0-4085-a418-4199f43b22ea admin admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:08:17.019704 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bf8c1be-77c0-4085-a418-4199f43b22ea admin admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:08:17.025073 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8cdebbcc-4c1c-4c06-a82d-c074be668e3e service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:08:17.026261 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b78a78b0-be9d-418a-af1e-c254c6e861a2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:08:17.026693 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8cdebbcc-4c1c-4c06-a82d-c074be668e3e service cinder] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:08:17.027287 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b78a78b0-be9d-418a-af1e-c254c6e861a2 admin admin] REQUEST_METHOD: `PUT` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:08:17.027637 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b78a78b0-be9d-418a-af1e-c254c6e861a2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:08:17.027983 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b78a78b0-be9d-418a-af1e-c254c6e861a2 admin admin] PATH_INFO: `/v3/projects/7bcd5c0750244995bb4894905f6546f9/users/960967bfbedb4e1aa6fdc61b1e85c427/roles/15db4aaca3264ae29bde38903ba43471` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:08:17.043046 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1015/1998] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 16:08:16 2022] GET /identity/v3/auth/tokens => generated 3311 bytes in 244 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 13 16:08:17.047720 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b78a78b0-be9d-418a-af1e-c254c6e861a2 admin admin] RBAC: Authorizing `identity:create_grant(project_id=7bcd5c0750244995bb4894905f6546f9, user_id=960967bfbedb4e1aa6fdc61b1e85c427, role_id=15db4aaca3264ae29bde38903ba43471)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:08:17.050240 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b78a78b0-be9d-418a-af1e-c254c6e861a2 admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:08:17.067723 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 984/1999] 10.222.0.50 () {60 vars in 1483 bytes} [Tue Sep 13 16:08:17 2022] PUT /identity/v3/projects/7bcd5c0750244995bb4894905f6546f9/users/960967bfbedb4e1aa6fdc61b1e85c427/roles/15db4aaca3264ae29bde38903ba43471 => generated 0 bytes in 61 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 16:08:17.079079 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-320fe623-44cb-4ade-8e1b-479491388a6a None None] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:08:17.079311 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-320fe623-44cb-4ade-8e1b-479491388a6a None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:08:17.079543 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-320fe623-44cb-4ade-8e1b-479491388a6a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:08:17.112879 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.auth.core [None req-320fe623-44cb-4ade-8e1b-479491388a6a None None] MFA Rules not processed for user `960967bfbedb4e1aa6fdc61b1e85c427`. Rule list: `[]` (Enabled: `True`). {{(pid=81437) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:08:17.114185 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-320fe623-44cb-4ade-8e1b-479491388a6a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:08:17.157647 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b78a78b0-be9d-418a-af1e-c254c6e861a2 admin admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:08:17.199257 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1016/2000] 10.222.0.50 () {58 vars in 958 bytes} [Tue Sep 13 16:08:17 2022] POST /identity/v3/auth/tokens => generated 3242 bytes in 122 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:08:17.201062 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-320fe623-44cb-4ade-8e1b-479491388a6a None None] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:08:17.262247 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b78a78b0-be9d-418a-af1e-c254c6e861a2 admin admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:08:17.289431 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fdbcceb-a62a-4169-8fbf-7361dca0f05f service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7fd6b5afd67b48f9b0f0597ccd4cf4df', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:08:17.290061 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fdbcceb-a62a-4169-8fbf-7361dca0f05f service cinder] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:08:17.290339 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fdbcceb-a62a-4169-8fbf-7361dca0f05f service cinder] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:08:17.290612 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fdbcceb-a62a-4169-8fbf-7361dca0f05f service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:08:17.292714 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-5fdbcceb-a62a-4169-8fbf-7361dca0f05f service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:08:17.330370 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-320fe623-44cb-4ade-8e1b-479491388a6a None None] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:08:17.338880 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83afbec0-22fe-40ee-ad35-004299ea8111 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:08:17.340090 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83afbec0-22fe-40ee-ad35-004299ea8111 admin admin] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:08:17.341383 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83afbec0-22fe-40ee-ad35-004299ea8111 admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:08:17.342044 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83afbec0-22fe-40ee-ad35-004299ea8111 admin admin] PATH_INFO: `/v3/projects` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:08:17.343347 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83afbec0-22fe-40ee-ad35-004299ea8111 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:08:17.347145 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83afbec0-22fe-40ee-ad35-004299ea8111 admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:08:17.347698 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:08:17.369476 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:08:17.376344 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fdbcceb-a62a-4169-8fbf-7361dca0f05f service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:08:17.377797 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fdbcceb-a62a-4169-8fbf-7361dca0f05f service cinder] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:08:17.395013 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 985/2001] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 16:08:17 2022] GET /identity/v3/auth/tokens => generated 3244 bytes in 236 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 16:08:17.411332 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1017/2002] 10.222.0.50 () {60 vars in 1153 bytes} [Tue Sep 13 16:08:17 2022] POST /identity/v3/projects => generated 365 bytes in 211 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 13 16:08:17.418018 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fdbcceb-a62a-4169-8fbf-7361dca0f05f service cinder] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:08:17.429430 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fdbcceb-a62a-4169-8fbf-7361dca0f05f service cinder] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:08:17.508828 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29503f89-22a1-45ea-9235-f3440cf8e8ee admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:08:17.509444 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29503f89-22a1-45ea-9235-f3440cf8e8ee admin admin] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:08:17.510020 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29503f89-22a1-45ea-9235-f3440cf8e8ee admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:08:17.510303 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29503f89-22a1-45ea-9235-f3440cf8e8ee admin admin] PATH_INFO: `/v3/users` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:08:17.511026 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29503f89-22a1-45ea-9235-f3440cf8e8ee admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:08:17.513601 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29503f89-22a1-45ea-9235-f3440cf8e8ee admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:08:17.539800 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 986/2003] 10.222.0.50 () {60 vars in 1144 bytes} [Tue Sep 13 16:08:17 2022] POST /identity/v3/users => generated 289 bytes in 122 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 16:08:17.544029 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83afbec0-22fe-40ee-ad35-004299ea8111 admin admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:08:17.556026 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83afbec0-22fe-40ee-ad35-004299ea8111 admin admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:08:17.561556 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ee31f26-8d99-4011-be01-bfdf5d889476 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:08:17.562180 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ee31f26-8d99-4011-be01-bfdf5d889476 admin admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:08:17.562464 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ee31f26-8d99-4011-be01-bfdf5d889476 admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:08:17.562739 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ee31f26-8d99-4011-be01-bfdf5d889476 admin admin] PATH_INFO: `/v3/roles` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:08:17.563432 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ee31f26-8d99-4011-be01-bfdf5d889476 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:08:17.565200 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ee31f26-8d99-4011-be01-bfdf5d889476 admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:08:17.578129 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1018/2004] 10.222.0.50 () {58 vars in 1122 bytes} [Tue Sep 13 16:08:17 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 16:08:17.582243 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29503f89-22a1-45ea-9235-f3440cf8e8ee admin admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:08:17.594555 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29503f89-22a1-45ea-9235-f3440cf8e8ee admin admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:08:17.599875 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56f09543-85c8-45ca-bbc6-290522994204 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:08:17.600725 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56f09543-85c8-45ca-bbc6-290522994204 admin admin] REQUEST_METHOD: `PUT` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:08:17.601004 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56f09543-85c8-45ca-bbc6-290522994204 admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:08:17.601340 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56f09543-85c8-45ca-bbc6-290522994204 admin admin] PATH_INFO: `/v3/projects/f041fd2f1f454f4cb355a5c84a3f7ae5/users/ee11b1b75f0f4499b1ab7092854ce99d/roles/031d4e1a23e34f71858fc4a6dfbd4ccb` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:08:17.622403 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56f09543-85c8-45ca-bbc6-290522994204 admin admin] RBAC: Authorizing `identity:create_grant(project_id=f041fd2f1f454f4cb355a5c84a3f7ae5, user_id=ee11b1b75f0f4499b1ab7092854ce99d, role_id=031d4e1a23e34f71858fc4a6dfbd4ccb)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:08:17.627247 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56f09543-85c8-45ca-bbc6-290522994204 admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:08:17.642066 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 987/2005] 10.222.0.50 () {60 vars in 1483 bytes} [Tue Sep 13 16:08:17 2022] PUT /identity/v3/projects/f041fd2f1f454f4cb355a5c84a3f7ae5/users/ee11b1b75f0f4499b1ab7092854ce99d/roles/031d4e1a23e34f71858fc4a6dfbd4ccb => generated 0 bytes in 60 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 16:08:17.646004 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ee31f26-8d99-4011-be01-bfdf5d889476 admin admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:08:17.659084 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ee31f26-8d99-4011-be01-bfdf5d889476 admin admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:08:17.743540 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52efb514-a885-4dc5-8df2-7f753d276789 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:08:17.744291 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52efb514-a885-4dc5-8df2-7f753d276789 admin admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:08:17.744633 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52efb514-a885-4dc5-8df2-7f753d276789 admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:08:17.745037 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52efb514-a885-4dc5-8df2-7f753d276789 admin admin] PATH_INFO: `/v3/roles` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:08:17.745702 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52efb514-a885-4dc5-8df2-7f753d276789 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:08:17.747429 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52efb514-a885-4dc5-8df2-7f753d276789 admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:08:17.767085 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1019/2006] 10.222.0.50 () {58 vars in 1122 bytes} [Tue Sep 13 16:08:17 2022] GET /identity/v3/roles => generated 1420 bytes in 122 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 16:08:17.772969 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56f09543-85c8-45ca-bbc6-290522994204 admin admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:08:17.792088 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56f09543-85c8-45ca-bbc6-290522994204 admin admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:08:17.796862 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55589d25-3f58-4d28-abc9-bc3b8be3df1f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:08:17.797540 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55589d25-3f58-4d28-abc9-bc3b8be3df1f admin admin] REQUEST_METHOD: `PUT` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:08:17.797839 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55589d25-3f58-4d28-abc9-bc3b8be3df1f admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:08:17.798103 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55589d25-3f58-4d28-abc9-bc3b8be3df1f admin admin] PATH_INFO: `/v3/projects/f041fd2f1f454f4cb355a5c84a3f7ae5/users/ee11b1b75f0f4499b1ab7092854ce99d/roles/031d4e1a23e34f71858fc4a6dfbd4ccb` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:08:17.800820 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55589d25-3f58-4d28-abc9-bc3b8be3df1f admin admin] RBAC: Authorizing `identity:create_grant(project_id=f041fd2f1f454f4cb355a5c84a3f7ae5, user_id=ee11b1b75f0f4499b1ab7092854ce99d, role_id=031d4e1a23e34f71858fc4a6dfbd4ccb)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:08:17.804111 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55589d25-3f58-4d28-abc9-bc3b8be3df1f admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:08:17.824527 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 988/2007] 10.222.0.50 () {60 vars in 1483 bytes} [Tue Sep 13 16:08:17 2022] PUT /identity/v3/projects/f041fd2f1f454f4cb355a5c84a3f7ae5/users/ee11b1b75f0f4499b1ab7092854ce99d/roles/031d4e1a23e34f71858fc4a6dfbd4ccb => generated 0 bytes in 46 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 16:08:17.829868 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91da7bdd-8631-4780-88f4-1a9ad5e4f216 None None] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:08:17.830166 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91da7bdd-8631-4780-88f4-1a9ad5e4f216 None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:08:17.830446 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91da7bdd-8631-4780-88f4-1a9ad5e4f216 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:08:17.860950 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.auth.core [None req-91da7bdd-8631-4780-88f4-1a9ad5e4f216 None None] MFA Rules not processed for user `ee11b1b75f0f4499b1ab7092854ce99d`. Rule list: `[]` (Enabled: `True`). {{(pid=81437) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:08:17.862154 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-91da7bdd-8631-4780-88f4-1a9ad5e4f216 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:08:17.929955 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1020/2008] 10.222.0.50 () {58 vars in 958 bytes} [Tue Sep 13 16:08:17 2022] POST /identity/v3/auth/tokens => generated 3309 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:08:17.941494 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55589d25-3f58-4d28-abc9-bc3b8be3df1f admin admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:08:17.998310 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55589d25-3f58-4d28-abc9-bc3b8be3df1f admin admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:08:18.003556 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86a578c6-a27f-4d2a-858d-f559bbf3aaec service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7fd6b5afd67b48f9b0f0597ccd4cf4df', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:08:18.004304 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86a578c6-a27f-4d2a-858d-f559bbf3aaec service cinder] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:08:18.004649 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86a578c6-a27f-4d2a-858d-f559bbf3aaec service cinder] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:08:18.004885 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86a578c6-a27f-4d2a-858d-f559bbf3aaec service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:08:18.015376 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86a578c6-a27f-4d2a-858d-f559bbf3aaec service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:08:18.016853 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86a578c6-a27f-4d2a-858d-f559bbf3aaec service cinder] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:08:18.027028 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 989/2009] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 16:08:17 2022] GET /identity/v3/auth/tokens => generated 3309 bytes in 86 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 13 16:08:18.105268 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91da7bdd-8631-4780-88f4-1a9ad5e4f216 None None] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:08:18.114737 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91da7bdd-8631-4780-88f4-1a9ad5e4f216 None None] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:08:18.118374 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-073319e3-11a4-41eb-af1d-83468ac674c9 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7fd6b5afd67b48f9b0f0597ccd4cf4df', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:08:18.119004 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-073319e3-11a4-41eb-af1d-83468ac674c9 service cinder] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:08:18.119287 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-073319e3-11a4-41eb-af1d-83468ac674c9 service cinder] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:08:18.119562 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-073319e3-11a4-41eb-af1d-83468ac674c9 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:08:18.178028 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-073319e3-11a4-41eb-af1d-83468ac674c9 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:08:18.179510 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-073319e3-11a4-41eb-af1d-83468ac674c9 service cinder] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:08:18.185016 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:08:18.189960 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1021/2010] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 16:08:18 2022] GET /identity/v3/auth/tokens => generated 3242 bytes in 85 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 16:08:21.023709 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86a578c6-a27f-4d2a-858d-f559bbf3aaec service cinder] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:08:21.085673 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:08:21.111208 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86a578c6-a27f-4d2a-858d-f559bbf3aaec service cinder] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:08:21.119738 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68c9b31d-fa2c-4749-a9c2-ddcac2cd921a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:08:21.120388 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68c9b31d-fa2c-4749-a9c2-ddcac2cd921a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:08:21.120671 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68c9b31d-fa2c-4749-a9c2-ddcac2cd921a admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:08:21.120950 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68c9b31d-fa2c-4749-a9c2-ddcac2cd921a admin admin] PATH_INFO: `/v3/users/960967bfbedb4e1aa6fdc61b1e85c427` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:08:21.122092 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68c9b31d-fa2c-4749-a9c2-ddcac2cd921a admin admin] RBAC: Authorizing `identity:delete_user(user_id=960967bfbedb4e1aa6fdc61b1e85c427)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:08:21.124053 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68c9b31d-fa2c-4749-a9c2-ddcac2cd921a admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:08:21.174710 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-68c9b31d-fa2c-4749-a9c2-ddcac2cd921a admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '960967bfbedb4e1aa6fdc61b1e85c427'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:08:21.184407 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-68c9b31d-fa2c-4749-a9c2-ddcac2cd921a admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '960967bfbedb4e1aa6fdc61b1e85c427'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:08:21.201932 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-68c9b31d-fa2c-4749-a9c2-ddcac2cd921a admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '960967bfbedb4e1aa6fdc61b1e85c427'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:08:21.201932 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-68c9b31d-fa2c-4749-a9c2-ddcac2cd921a admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '960967bfbedb4e1aa6fdc61b1e85c427'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:08:21.208851 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-68c9b31d-fa2c-4749-a9c2-ddcac2cd921a admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '960967bfbedb4e1aa6fdc61b1e85c427'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:08:21.215644 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 990/2011] 10.222.0.50 () {58 vars in 1224 bytes} [Tue Sep 13 16:08:21 2022] DELETE /identity/v3/users/960967bfbedb4e1aa6fdc61b1e85c427 => generated 0 bytes in 192 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 16:08:21.222075 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0d73c1d-0d80-4f34-8568-898c06f321c0 None None] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:08:21.222075 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0d73c1d-0d80-4f34-8568-898c06f321c0 None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:08:21.222075 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0d73c1d-0d80-4f34-8568-898c06f321c0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:08:21.242812 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.auth.core [None req-c0d73c1d-0d80-4f34-8568-898c06f321c0 None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81437) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:08:21.244249 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-c0d73c1d-0d80-4f34-8568-898c06f321c0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:08:21.316499 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1022/2012] 10.222.0.50 () {58 vars in 958 bytes} [Tue Sep 13 16:08:21 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:08:21.326844 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68c9b31d-fa2c-4749-a9c2-ddcac2cd921a admin admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:08:21.328314 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-68c9b31d-fa2c-4749-a9c2-ddcac2cd921a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:08:21.397947 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68c9b31d-fa2c-4749-a9c2-ddcac2cd921a admin admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:08:21.401894 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-417b5875-7836-4f07-8553-14f4636c6c13 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f436f07939d4855a21b195bc7566244', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:08:21.402582 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-417b5875-7836-4f07-8553-14f4636c6c13 service neutron] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:08:21.402968 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-417b5875-7836-4f07-8553-14f4636c6c13 service neutron] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:08:21.403314 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-417b5875-7836-4f07-8553-14f4636c6c13 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:08:21.416967 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-417b5875-7836-4f07-8553-14f4636c6c13 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:08:21.418419 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-417b5875-7836-4f07-8553-14f4636c6c13 service neutron] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:08:21.428725 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 991/2013] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 16:08:21 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 102 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 16:08:21.708317 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0d73c1d-0d80-4f34-8568-898c06f321c0 None None] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:08:21.710989 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-c0d73c1d-0d80-4f34-8568-898c06f321c0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:08:21.726941 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0d73c1d-0d80-4f34-8568-898c06f321c0 None None] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:08:21.732371 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af974eae-56c8-426a-8892-5188ce886378 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:08:21.733185 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af974eae-56c8-426a-8892-5188ce886378 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:08:21.733586 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af974eae-56c8-426a-8892-5188ce886378 admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:08:21.733952 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af974eae-56c8-426a-8892-5188ce886378 admin admin] PATH_INFO: `/v3/projects/7bcd5c0750244995bb4894905f6546f9` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:08:21.735245 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af974eae-56c8-426a-8892-5188ce886378 admin admin] RBAC: Authorizing `identity:delete_project(project_id=7bcd5c0750244995bb4894905f6546f9)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:08:21.737469 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af974eae-56c8-426a-8892-5188ce886378 admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:08:21.790764 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-af974eae-56c8-426a-8892-5188ce886378 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '7bcd5c0750244995bb4894905f6546f9', 'request_id': 'req-af974eae-56c8-426a-8892-5188ce886378'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:08:21.797134 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-af974eae-56c8-426a-8892-5188ce886378 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '7bcd5c0750244995bb4894905f6546f9', 'request_id': 'req-af974eae-56c8-426a-8892-5188ce886378'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:08:21.804110 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-af974eae-56c8-426a-8892-5188ce886378 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '7bcd5c0750244995bb4894905f6546f9', 'request_id': 'req-af974eae-56c8-426a-8892-5188ce886378'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:08:21.814123 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-af974eae-56c8-426a-8892-5188ce886378 admin admin] The token cache is being invalidate because project 7bcd5c0750244995bb4894905f6546f9 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81437) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 13 16:08:21.814407 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-af974eae-56c8-426a-8892-5188ce886378 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:08:21.815067 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-af974eae-56c8-426a-8892-5188ce886378 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:08:21.816550 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1023/2014] 10.222.0.50 () {58 vars in 1233 bytes} [Tue Sep 13 16:08:21 2022] DELETE /identity/v3/projects/7bcd5c0750244995bb4894905f6546f9 => generated 0 bytes in 109 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 13 16:08:21.821293 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-417b5875-7836-4f07-8553-14f4636c6c13 service neutron] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:08:21.831190 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-417b5875-7836-4f07-8553-14f4636c6c13 service neutron] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:08:21.883364 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:08:21.890423 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-435ec65c-6e92-4b57-93b3-6b5d6a5c8776 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:08:21.891161 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-435ec65c-6e92-4b57-93b3-6b5d6a5c8776 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:08:21.891604 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-435ec65c-6e92-4b57-93b3-6b5d6a5c8776 admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:08:21.891893 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-435ec65c-6e92-4b57-93b3-6b5d6a5c8776 admin admin] PATH_INFO: `/v3/users/ee11b1b75f0f4499b1ab7092854ce99d` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:08:21.893103 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-435ec65c-6e92-4b57-93b3-6b5d6a5c8776 admin admin] RBAC: Authorizing `identity:delete_user(user_id=ee11b1b75f0f4499b1ab7092854ce99d)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:08:21.895459 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-435ec65c-6e92-4b57-93b3-6b5d6a5c8776 admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:08:21.943034 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-435ec65c-6e92-4b57-93b3-6b5d6a5c8776 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'ee11b1b75f0f4499b1ab7092854ce99d'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:08:21.950415 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-435ec65c-6e92-4b57-93b3-6b5d6a5c8776 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'ee11b1b75f0f4499b1ab7092854ce99d'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:08:21.962507 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-435ec65c-6e92-4b57-93b3-6b5d6a5c8776 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'ee11b1b75f0f4499b1ab7092854ce99d'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:08:21.963204 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-435ec65c-6e92-4b57-93b3-6b5d6a5c8776 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'ee11b1b75f0f4499b1ab7092854ce99d'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:08:21.969945 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-435ec65c-6e92-4b57-93b3-6b5d6a5c8776 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'ee11b1b75f0f4499b1ab7092854ce99d'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:08:21.976705 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 992/2015] 10.222.0.50 () {58 vars in 1224 bytes} [Tue Sep 13 16:08:21 2022] DELETE /identity/v3/users/ee11b1b75f0f4499b1ab7092854ce99d => generated 0 bytes in 156 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 16:08:22.270484 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af974eae-56c8-426a-8892-5188ce886378 admin admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:08:22.272480 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-af974eae-56c8-426a-8892-5188ce886378 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:08:22.285472 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af974eae-56c8-426a-8892-5188ce886378 admin admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:08:22.360668 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-731e6478-c198-425d-a026-c1866c39d356 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:08:22.361317 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-731e6478-c198-425d-a026-c1866c39d356 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:08:22.361616 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-731e6478-c198-425d-a026-c1866c39d356 admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:08:22.361895 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-731e6478-c198-425d-a026-c1866c39d356 admin admin] PATH_INFO: `/v3/projects/f041fd2f1f454f4cb355a5c84a3f7ae5` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:08:22.363073 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-731e6478-c198-425d-a026-c1866c39d356 admin admin] RBAC: Authorizing `identity:delete_project(project_id=f041fd2f1f454f4cb355a5c84a3f7ae5)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:08:22.365357 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-731e6478-c198-425d-a026-c1866c39d356 admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:08:22.397186 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=91,keystone:DELETE=8,keystone:INSERT=1 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:08:22.417993 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-731e6478-c198-425d-a026-c1866c39d356 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'f041fd2f1f454f4cb355a5c84a3f7ae5', 'request_id': 'req-731e6478-c198-425d-a026-c1866c39d356'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:08:22.425358 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-731e6478-c198-425d-a026-c1866c39d356 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'f041fd2f1f454f4cb355a5c84a3f7ae5', 'request_id': 'req-731e6478-c198-425d-a026-c1866c39d356'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:08:22.431892 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-731e6478-c198-425d-a026-c1866c39d356 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'f041fd2f1f454f4cb355a5c84a3f7ae5', 'request_id': 'req-731e6478-c198-425d-a026-c1866c39d356'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:08:22.443436 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-731e6478-c198-425d-a026-c1866c39d356 admin admin] The token cache is being invalidate because project f041fd2f1f454f4cb355a5c84a3f7ae5 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81437) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 13 16:08:22.443798 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-731e6478-c198-425d-a026-c1866c39d356 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:08:22.444174 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-435ec65c-6e92-4b57-93b3-6b5d6a5c8776 admin admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:08:22.444618 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-731e6478-c198-425d-a026-c1866c39d356 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:08:22.445823 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-435ec65c-6e92-4b57-93b3-6b5d6a5c8776 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:08:22.446055 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1024/2016] 10.222.0.50 () {58 vars in 1233 bytes} [Tue Sep 13 16:08:22 2022] DELETE /identity/v3/projects/f041fd2f1f454f4cb355a5c84a3f7ae5 => generated 0 bytes in 176 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 16:08:22.450464 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2b962db-cb1d-4ec0-a7a1-e3af2b90437f None None] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:08:22.450754 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2b962db-cb1d-4ec0-a7a1-e3af2b90437f None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:08:22.451104 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2b962db-cb1d-4ec0-a7a1-e3af2b90437f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:08:22.459638 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-435ec65c-6e92-4b57-93b3-6b5d6a5c8776 admin admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:08:22.471759 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.auth.core [None req-d2b962db-cb1d-4ec0-a7a1-e3af2b90437f None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81437) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:08:22.472986 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-d2b962db-cb1d-4ec0-a7a1-e3af2b90437f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:08:22.531973 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b92d64d3-af75-49c3-92d4-5a9406e55396 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:08:22.532620 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b92d64d3-af75-49c3-92d4-5a9406e55396 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:08:22.532901 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b92d64d3-af75-49c3-92d4-5a9406e55396 admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:08:22.533175 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b92d64d3-af75-49c3-92d4-5a9406e55396 admin admin] PATH_INFO: `/v3/users/f7997f715f564aa0b32c5f9d383264a6` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:08:22.534481 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b92d64d3-af75-49c3-92d4-5a9406e55396 admin admin] RBAC: Authorizing `identity:delete_user(user_id=f7997f715f564aa0b32c5f9d383264a6)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:08:22.537015 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b92d64d3-af75-49c3-92d4-5a9406e55396 admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:08:22.546190 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1025/2017] 10.222.0.50 () {58 vars in 958 bytes} [Tue Sep 13 16:08:22 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 13 16:08:22.553974 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44a945b6-e693-43ca-aa2d-a3e895d543b5 None None] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:08:22.554265 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44a945b6-e693-43ca-aa2d-a3e895d543b5 None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:08:22.554544 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44a945b6-e693-43ca-aa2d-a3e895d543b5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:08:22.575159 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.auth.core [None req-44a945b6-e693-43ca-aa2d-a3e895d543b5 None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81437) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:08:22.576371 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-44a945b6-e693-43ca-aa2d-a3e895d543b5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:08:22.587420 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1026/2018] 10.222.0.50 () {58 vars in 958 bytes} [Tue Sep 13 16:08:22 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 13 16:08:22.589317 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-b92d64d3-af75-49c3-92d4-5a9406e55396 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'f7997f715f564aa0b32c5f9d383264a6'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:08:22.590835 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44a945b6-e693-43ca-aa2d-a3e895d543b5 None None] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:08:22.604571 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44a945b6-e693-43ca-aa2d-a3e895d543b5 None None] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:08:22.609755 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-b92d64d3-af75-49c3-92d4-5a9406e55396 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'f7997f715f564aa0b32c5f9d383264a6'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:08:22.610673 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb7fc4c5-2186-4ed9-a704-3792337fea97 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:08:22.611415 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb7fc4c5-2186-4ed9-a704-3792337fea97 admin admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:08:22.611748 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb7fc4c5-2186-4ed9-a704-3792337fea97 admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:08:22.612026 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb7fc4c5-2186-4ed9-a704-3792337fea97 admin admin] PATH_INFO: `/v3/domains` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:08:22.612738 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb7fc4c5-2186-4ed9-a704-3792337fea97 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81437) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 16:08:22.613051 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb7fc4c5-2186-4ed9-a704-3792337fea97 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:08:22.614782 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb7fc4c5-2186-4ed9-a704-3792337fea97 admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:08:22.623857 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-b92d64d3-af75-49c3-92d4-5a9406e55396 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'f7997f715f564aa0b32c5f9d383264a6'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:08:22.624598 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-b92d64d3-af75-49c3-92d4-5a9406e55396 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'f7997f715f564aa0b32c5f9d383264a6'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:08:22.632593 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1027/2019] 10.222.0.50 () {58 vars in 1153 bytes} [Tue Sep 13 16:08:22 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 16:08:22.632741 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=78,keystone:DELETE=20,keystone:INSERT=2 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:08:22.632938 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-b92d64d3-af75-49c3-92d4-5a9406e55396 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'f7997f715f564aa0b32c5f9d383264a6'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:08:22.633651 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb7fc4c5-2186-4ed9-a704-3792337fea97 admin admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:08:22.635393 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-bb7fc4c5-2186-4ed9-a704-3792337fea97 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:08:22.641335 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 993/2020] 10.222.0.50 () {58 vars in 1224 bytes} [Tue Sep 13 16:08:22 2022] DELETE /identity/v3/users/f7997f715f564aa0b32c5f9d383264a6 => generated 0 bytes in 198 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 13 16:08:22.647083 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-193cf324-7dfb-4af4-be7b-54e74220c300 None None] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:08:22.647506 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-193cf324-7dfb-4af4-be7b-54e74220c300 None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:08:22.647904 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-193cf324-7dfb-4af4-be7b-54e74220c300 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:08:22.649485 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb7fc4c5-2186-4ed9-a704-3792337fea97 admin admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:08:22.667534 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.auth.core [None req-193cf324-7dfb-4af4-be7b-54e74220c300 None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81438) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:08:22.668871 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-193cf324-7dfb-4af4-be7b-54e74220c300 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:08:22.713604 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7aba9d6a-3f53-4dc2-9e97-b2ad1a261372 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:08:22.714235 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7aba9d6a-3f53-4dc2-9e97-b2ad1a261372 admin admin] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:08:22.715402 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7aba9d6a-3f53-4dc2-9e97-b2ad1a261372 admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:08:22.715854 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7aba9d6a-3f53-4dc2-9e97-b2ad1a261372 admin admin] PATH_INFO: `/v3/projects` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:08:22.716550 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7aba9d6a-3f53-4dc2-9e97-b2ad1a261372 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:08:22.719312 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7aba9d6a-3f53-4dc2-9e97-b2ad1a261372 admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:08:22.740639 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 994/2021] 10.222.0.50 () {58 vars in 958 bytes} [Tue Sep 13 16:08:22 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 13 16:08:22.751386 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1028/2022] 10.222.0.50 () {60 vars in 1153 bytes} [Tue Sep 13 16:08:22 2022] POST /identity/v3/projects => generated 389 bytes in 115 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 16:08:22.751535 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-193cf324-7dfb-4af4-be7b-54e74220c300 None None] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:08:22.751870 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-193cf324-7dfb-4af4-be7b-54e74220c300 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:08:22.752339 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7aba9d6a-3f53-4dc2-9e97-b2ad1a261372 admin admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:08:22.774623 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:DELETE=6,keystone:SELECT=92,keystone:INSERT=2 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:08:22.816229 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-193cf324-7dfb-4af4-be7b-54e74220c300 None None] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:08:22.838304 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7aba9d6a-3f53-4dc2-9e97-b2ad1a261372 admin admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:08:22.843699 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-320024d4-a614-4205-ab01-40d0d4e62ada admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:08:22.844369 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-320024d4-a614-4205-ab01-40d0d4e62ada admin admin] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:08:22.845028 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-320024d4-a614-4205-ab01-40d0d4e62ada admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:08:22.845618 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-320024d4-a614-4205-ab01-40d0d4e62ada admin admin] PATH_INFO: `/v3/users` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:08:22.846304 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-320024d4-a614-4205-ab01-40d0d4e62ada admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:08:22.850387 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-320024d4-a614-4205-ab01-40d0d4e62ada admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:08:22.876306 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1029/2023] 10.222.0.50 () {60 vars in 1144 bytes} [Tue Sep 13 16:08:22 2022] POST /identity/v3/users => generated 295 bytes in 124 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 13 16:08:22.883091 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-320024d4-a614-4205-ab01-40d0d4e62ada admin admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:08:22.884590 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18e91878-9f68-4895-9010-227a64c57827 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f436f07939d4855a21b195bc7566244', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:08:22.885237 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18e91878-9f68-4895-9010-227a64c57827 service neutron] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:08:22.885510 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18e91878-9f68-4895-9010-227a64c57827 service neutron] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:08:22.885781 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18e91878-9f68-4895-9010-227a64c57827 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:08:22.894332 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-320024d4-a614-4205-ab01-40d0d4e62ada admin admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:08:22.895552 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18e91878-9f68-4895-9010-227a64c57827 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:08:22.898211 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18e91878-9f68-4895-9010-227a64c57827 service neutron] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:08:22.900322 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc25145e-7bb2-4c50-a2ab-26a562ac4de6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:08:22.900686 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc25145e-7bb2-4c50-a2ab-26a562ac4de6 admin admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:08:22.900997 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc25145e-7bb2-4c50-a2ab-26a562ac4de6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:08:22.901273 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc25145e-7bb2-4c50-a2ab-26a562ac4de6 admin admin] PATH_INFO: `/v3/roles` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:08:22.901923 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc25145e-7bb2-4c50-a2ab-26a562ac4de6 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:08:22.904000 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc25145e-7bb2-4c50-a2ab-26a562ac4de6 admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:08:22.908960 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 995/2024] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 16:08:22 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 159 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 16:08:22.915243 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1030/2025] 10.222.0.50 () {58 vars in 1122 bytes} [Tue Sep 13 16:08:22 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 16:08:22.920065 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18e91878-9f68-4895-9010-227a64c57827 service neutron] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:08:22.932949 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18e91878-9f68-4895-9010-227a64c57827 service neutron] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:08:22.942707 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe512294-5d1d-4d96-ab6d-d664d87a4f94 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:08:22.943505 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe512294-5d1d-4d96-ab6d-d664d87a4f94 admin admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:08:22.943851 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe512294-5d1d-4d96-ab6d-d664d87a4f94 admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:08:22.944192 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe512294-5d1d-4d96-ab6d-d664d87a4f94 admin admin] PATH_INFO: `/v3/roles` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:08:22.945884 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe512294-5d1d-4d96-ab6d-d664d87a4f94 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:08:22.947694 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe512294-5d1d-4d96-ab6d-d664d87a4f94 admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:08:22.963737 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 996/2026] 10.222.0.50 () {58 vars in 1122 bytes} [Tue Sep 13 16:08:22 2022] GET /identity/v3/roles => generated 1420 bytes in 44 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 16:08:22.969027 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc25145e-7bb2-4c50-a2ab-26a562ac4de6 admin admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:08:22.984353 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc25145e-7bb2-4c50-a2ab-26a562ac4de6 admin admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:08:22.998262 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00d925b8-8934-4e9c-b620-8681d56a92ce admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:08:22.999033 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00d925b8-8934-4e9c-b620-8681d56a92ce admin admin] REQUEST_METHOD: `PUT` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:08:22.999322 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00d925b8-8934-4e9c-b620-8681d56a92ce admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:08:22.999592 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00d925b8-8934-4e9c-b620-8681d56a92ce admin admin] PATH_INFO: `/v3/projects/1dfc0424cf874568aeb9edd87d6905ec/users/275eec5bdb5340aba09304a01868a62e/roles/15db4aaca3264ae29bde38903ba43471` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:08:23.025912 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00d925b8-8934-4e9c-b620-8681d56a92ce admin admin] RBAC: Authorizing `identity:create_grant(project_id=1dfc0424cf874568aeb9edd87d6905ec, user_id=275eec5bdb5340aba09304a01868a62e, role_id=15db4aaca3264ae29bde38903ba43471)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:08:23.028795 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00d925b8-8934-4e9c-b620-8681d56a92ce admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:08:23.045911 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1031/2027] 10.222.0.50 () {60 vars in 1483 bytes} [Tue Sep 13 16:08:22 2022] PUT /identity/v3/projects/1dfc0424cf874568aeb9edd87d6905ec/users/275eec5bdb5340aba09304a01868a62e/roles/15db4aaca3264ae29bde38903ba43471 => generated 0 bytes in 78 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 16:08:23.056527 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82438724-6d76-4ec3-9a07-451b8f722531 None None] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:08:23.056818 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82438724-6d76-4ec3-9a07-451b8f722531 None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:08:23.057155 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82438724-6d76-4ec3-9a07-451b8f722531 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:08:23.091841 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.auth.core [None req-82438724-6d76-4ec3-9a07-451b8f722531 None None] MFA Rules not processed for user `275eec5bdb5340aba09304a01868a62e`. Rule list: `[]` (Enabled: `True`). {{(pid=81438) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:08:23.092471 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-82438724-6d76-4ec3-9a07-451b8f722531 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:08:23.136833 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:08:23.170353 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 997/2028] 10.222.0.50 () {58 vars in 958 bytes} [Tue Sep 13 16:08:23 2022] POST /identity/v3/auth/tokens => generated 3266 bytes in 115 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:08:23.178666 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00d925b8-8934-4e9c-b620-8681d56a92ce admin admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:08:23.245738 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00d925b8-8934-4e9c-b620-8681d56a92ce admin admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:08:23.250792 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a968122-5025-4449-b531-d682965c4e2a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:08:23.251009 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a968122-5025-4449-b531-d682965c4e2a admin admin] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:08:23.251290 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a968122-5025-4449-b531-d682965c4e2a admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:08:23.251665 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a968122-5025-4449-b531-d682965c4e2a admin admin] PATH_INFO: `/v3/projects` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:08:23.252402 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a968122-5025-4449-b531-d682965c4e2a admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:08:23.254321 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a968122-5025-4449-b531-d682965c4e2a admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:08:23.285087 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1032/2029] 10.222.0.50 () {60 vars in 1153 bytes} [Tue Sep 13 16:08:23 2022] POST /identity/v3/projects => generated 389 bytes in 111 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 13 16:08:23.290684 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82438724-6d76-4ec3-9a07-451b8f722531 None None] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:08:23.307664 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82438724-6d76-4ec3-9a07-451b8f722531 None None] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:08:23.396634 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87fedb73-44e0-46c0-9098-d40eb02ef5f5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:08:23.396980 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a968122-5025-4449-b531-d682965c4e2a admin admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:08:23.397180 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87fedb73-44e0-46c0-9098-d40eb02ef5f5 admin admin] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:08:23.397465 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87fedb73-44e0-46c0-9098-d40eb02ef5f5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:08:23.397736 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87fedb73-44e0-46c0-9098-d40eb02ef5f5 admin admin] PATH_INFO: `/v3/users` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:08:23.398486 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87fedb73-44e0-46c0-9098-d40eb02ef5f5 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:08:23.399833 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-7a968122-5025-4449-b531-d682965c4e2a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:08:23.400670 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87fedb73-44e0-46c0-9098-d40eb02ef5f5 admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:08:23.422349 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a968122-5025-4449-b531-d682965c4e2a admin admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:08:23.430514 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa8b7914-bac0-4c3e-93a0-aee4f5a05fbc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:08:23.431571 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa8b7914-bac0-4c3e-93a0-aee4f5a05fbc admin admin] REQUEST_METHOD: `DELETE` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:08:23.432236 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa8b7914-bac0-4c3e-93a0-aee4f5a05fbc admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:08:23.432692 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa8b7914-bac0-4c3e-93a0-aee4f5a05fbc admin admin] PATH_INFO: `/v3/projects/e9d2efd0f85c48ae8dd78b3a68daedff` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:08:23.440039 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa8b7914-bac0-4c3e-93a0-aee4f5a05fbc admin admin] RBAC: Authorizing `identity:delete_project(project_id=e9d2efd0f85c48ae8dd78b3a68daedff)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:08:23.442042 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa8b7914-bac0-4c3e-93a0-aee4f5a05fbc admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:08:23.461353 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 998/2030] 10.222.0.50 () {60 vars in 1144 bytes} [Tue Sep 13 16:08:23 2022] POST /identity/v3/users => generated 301 bytes in 168 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 16:08:23.464419 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87fedb73-44e0-46c0-9098-d40eb02ef5f5 admin admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:08:23.479555 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87fedb73-44e0-46c0-9098-d40eb02ef5f5 admin admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:08:23.486122 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae7063aa-b3a7-4021-9e14-598f59fdcac1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:08:23.487205 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae7063aa-b3a7-4021-9e14-598f59fdcac1 admin admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:08:23.487943 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae7063aa-b3a7-4021-9e14-598f59fdcac1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:08:23.488375 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae7063aa-b3a7-4021-9e14-598f59fdcac1 admin admin] PATH_INFO: `/v3/roles` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:08:23.489169 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae7063aa-b3a7-4021-9e14-598f59fdcac1 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:08:23.491310 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae7063aa-b3a7-4021-9e14-598f59fdcac1 admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:08:23.512763 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 999/2031] 10.222.0.50 () {58 vars in 1122 bytes} [Tue Sep 13 16:08:23 2022] GET /identity/v3/roles => generated 1420 bytes in 46 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 16:08:23.514000 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae7063aa-b3a7-4021-9e14-598f59fdcac1 admin admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:08:23.525583 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae7063aa-b3a7-4021-9e14-598f59fdcac1 admin admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:08:23.532727 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-aa8b7914-bac0-4c3e-93a0-aee4f5a05fbc admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'e9d2efd0f85c48ae8dd78b3a68daedff', 'request_id': 'req-aa8b7914-bac0-4c3e-93a0-aee4f5a05fbc'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:08:23.539861 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=89,keystone:INSERT=6,keystone:DELETE=5 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:08:23.550475 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-aa8b7914-bac0-4c3e-93a0-aee4f5a05fbc admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'e9d2efd0f85c48ae8dd78b3a68daedff', 'request_id': 'req-aa8b7914-bac0-4c3e-93a0-aee4f5a05fbc'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:08:23.563550 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-aa8b7914-bac0-4c3e-93a0-aee4f5a05fbc admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'e9d2efd0f85c48ae8dd78b3a68daedff', 'request_id': 'req-aa8b7914-bac0-4c3e-93a0-aee4f5a05fbc'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:08:23.582552 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-aa8b7914-bac0-4c3e-93a0-aee4f5a05fbc admin admin] The token cache is being invalidate because project e9d2efd0f85c48ae8dd78b3a68daedff was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81437) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 13 16:08:23.583895 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-aa8b7914-bac0-4c3e-93a0-aee4f5a05fbc admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:08:23.584939 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-aa8b7914-bac0-4c3e-93a0-aee4f5a05fbc admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:08:23.586877 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1033/2032] 10.222.0.50 () {58 vars in 1233 bytes} [Tue Sep 13 16:08:23 2022] DELETE /identity/v3/projects/e9d2efd0f85c48ae8dd78b3a68daedff => generated 0 bytes in 191 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 13 16:08:23.590529 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa8b7914-bac0-4c3e-93a0-aee4f5a05fbc admin admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:08:23.592118 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-aa8b7914-bac0-4c3e-93a0-aee4f5a05fbc admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:08:23.600618 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-188ddd9a-1acc-4074-9708-4bdc3279c3d6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:08:23.601282 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-188ddd9a-1acc-4074-9708-4bdc3279c3d6 admin admin] REQUEST_METHOD: `PUT` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:08:23.601561 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-188ddd9a-1acc-4074-9708-4bdc3279c3d6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:08:23.601868 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-188ddd9a-1acc-4074-9708-4bdc3279c3d6 admin admin] PATH_INFO: `/v3/projects/c8fc3b46ad7b44108a730a9c1dfe6b6e/users/3ca4bc81953441a5b94d18556a1a9621/roles/031d4e1a23e34f71858fc4a6dfbd4ccb` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:08:23.618990 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-188ddd9a-1acc-4074-9708-4bdc3279c3d6 admin admin] RBAC: Authorizing `identity:create_grant(project_id=c8fc3b46ad7b44108a730a9c1dfe6b6e, user_id=3ca4bc81953441a5b94d18556a1a9621, role_id=031d4e1a23e34f71858fc4a6dfbd4ccb)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:08:23.621656 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-188ddd9a-1acc-4074-9708-4bdc3279c3d6 admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:08:23.648459 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1000/2033] 10.222.0.50 () {60 vars in 1483 bytes} [Tue Sep 13 16:08:23 2022] PUT /identity/v3/projects/c8fc3b46ad7b44108a730a9c1dfe6b6e/users/3ca4bc81953441a5b94d18556a1a9621/roles/031d4e1a23e34f71858fc4a6dfbd4ccb => generated 0 bytes in 135 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 16:08:23.652620 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-188ddd9a-1acc-4074-9708-4bdc3279c3d6 admin admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:08:23.654524 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-188ddd9a-1acc-4074-9708-4bdc3279c3d6 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:08:23.682423 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa8b7914-bac0-4c3e-93a0-aee4f5a05fbc admin admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:08:23.699950 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:08:23.734714 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-188ddd9a-1acc-4074-9708-4bdc3279c3d6 admin admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:08:23.739951 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98d4a897-1ffa-4cbb-a828-2eed6e4e9854 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:08:23.740879 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98d4a897-1ffa-4cbb-a828-2eed6e4e9854 admin admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:08:23.741156 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98d4a897-1ffa-4cbb-a828-2eed6e4e9854 admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:08:23.741434 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98d4a897-1ffa-4cbb-a828-2eed6e4e9854 admin admin] PATH_INFO: `/v3/roles` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:08:23.742074 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98d4a897-1ffa-4cbb-a828-2eed6e4e9854 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:08:23.743717 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98d4a897-1ffa-4cbb-a828-2eed6e4e9854 admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:08:23.746965 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1355659-62cc-499b-a394-e81df2dc6484 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:08:23.747614 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1355659-62cc-499b-a394-e81df2dc6484 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:08:23.747897 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1355659-62cc-499b-a394-e81df2dc6484 admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:08:23.748223 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1355659-62cc-499b-a394-e81df2dc6484 admin admin] PATH_INFO: `/v3/users/6036198a0936436bbb563def1464dc04` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:08:23.749881 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1355659-62cc-499b-a394-e81df2dc6484 admin admin] RBAC: Authorizing `identity:delete_user(user_id=6036198a0936436bbb563def1464dc04)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:08:23.752045 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1355659-62cc-499b-a394-e81df2dc6484 admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:08:23.756537 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1001/2034] 10.222.0.50 () {58 vars in 1122 bytes} [Tue Sep 13 16:08:23 2022] GET /identity/v3/roles => generated 1420 bytes in 104 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 16:08:23.760215 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98d4a897-1ffa-4cbb-a828-2eed6e4e9854 admin admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:08:23.772804 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98d4a897-1ffa-4cbb-a828-2eed6e4e9854 admin admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:08:23.790668 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a32feb1-a14d-40ef-88c7-def06b66adfe admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:08:23.791900 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a32feb1-a14d-40ef-88c7-def06b66adfe admin admin] REQUEST_METHOD: `PUT` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:08:23.792518 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a32feb1-a14d-40ef-88c7-def06b66adfe admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:08:23.792872 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a32feb1-a14d-40ef-88c7-def06b66adfe admin admin] PATH_INFO: `/v3/projects/c8fc3b46ad7b44108a730a9c1dfe6b6e/users/3ca4bc81953441a5b94d18556a1a9621/roles/031d4e1a23e34f71858fc4a6dfbd4ccb` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:08:23.796311 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a32feb1-a14d-40ef-88c7-def06b66adfe admin admin] RBAC: Authorizing `identity:create_grant(project_id=c8fc3b46ad7b44108a730a9c1dfe6b6e, user_id=3ca4bc81953441a5b94d18556a1a9621, role_id=031d4e1a23e34f71858fc4a6dfbd4ccb)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:08:23.800011 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a32feb1-a14d-40ef-88c7-def06b66adfe admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:08:23.833542 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1002/2035] 10.222.0.50 () {60 vars in 1483 bytes} [Tue Sep 13 16:08:23 2022] PUT /identity/v3/projects/c8fc3b46ad7b44108a730a9c1dfe6b6e/users/3ca4bc81953441a5b94d18556a1a9621/roles/031d4e1a23e34f71858fc4a6dfbd4ccb => generated 0 bytes in 62 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 13 16:08:23.836438 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d54d8a9d-7809-4e23-8ca5-33b9eba6ea8e None None] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:08:23.836798 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d54d8a9d-7809-4e23-8ca5-33b9eba6ea8e None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:08:23.837195 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d54d8a9d-7809-4e23-8ca5-33b9eba6ea8e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:08:23.849915 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-a1355659-62cc-499b-a394-e81df2dc6484 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '6036198a0936436bbb563def1464dc04'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:08:23.859516 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-a1355659-62cc-499b-a394-e81df2dc6484 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '6036198a0936436bbb563def1464dc04'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:08:23.875603 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-a1355659-62cc-499b-a394-e81df2dc6484 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '6036198a0936436bbb563def1464dc04'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:08:23.876302 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-a1355659-62cc-499b-a394-e81df2dc6484 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '6036198a0936436bbb563def1464dc04'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:08:23.876782 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.auth.core [None req-d54d8a9d-7809-4e23-8ca5-33b9eba6ea8e None None] MFA Rules not processed for user `3ca4bc81953441a5b94d18556a1a9621`. Rule list: `[]` (Enabled: `True`). {{(pid=81438) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:08:23.878603 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-d54d8a9d-7809-4e23-8ca5-33b9eba6ea8e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:08:23.890939 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-a1355659-62cc-499b-a394-e81df2dc6484 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '6036198a0936436bbb563def1464dc04'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:08:23.901288 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1034/2036] 10.222.0.50 () {58 vars in 1224 bytes} [Tue Sep 13 16:08:23 2022] DELETE /identity/v3/users/6036198a0936436bbb563def1464dc04 => generated 0 bytes in 308 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 16:08:23.973546 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1003/2037] 10.222.0.50 () {58 vars in 958 bytes} [Tue Sep 13 16:08:23 2022] POST /identity/v3/auth/tokens => generated 3333 bytes in 139 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:08:23.985185 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1355659-62cc-499b-a394-e81df2dc6484 admin admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:08:23.986676 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-a1355659-62cc-499b-a394-e81df2dc6484 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:08:23.999472 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=87,keystone:DELETE=12,keystone:INSERT=1 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:08:24.070336 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1355659-62cc-499b-a394-e81df2dc6484 admin admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:08:24.076456 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-535be690-d929-4842-8f7c-402cd3264e4b service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7fd6b5afd67b48f9b0f0597ccd4cf4df', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:08:24.077359 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-535be690-d929-4842-8f7c-402cd3264e4b service cinder] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:08:24.077953 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-535be690-d929-4842-8f7c-402cd3264e4b service cinder] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:08:24.078378 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-535be690-d929-4842-8f7c-402cd3264e4b service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:08:24.173881 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-535be690-d929-4842-8f7c-402cd3264e4b service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:08:24.175414 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-535be690-d929-4842-8f7c-402cd3264e4b service cinder] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:08:24.187227 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1035/2038] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 16:08:23 2022] GET /identity/v3/auth/tokens => generated 3333 bytes in 203 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 16:08:24.373288 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d54d8a9d-7809-4e23-8ca5-33b9eba6ea8e None None] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:08:24.374851 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-d54d8a9d-7809-4e23-8ca5-33b9eba6ea8e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:08:24.456513 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d54d8a9d-7809-4e23-8ca5-33b9eba6ea8e None None] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:08:24.529282 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:08:24.548039 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-007b99f9-fdc7-4597-9a86-f8f888d3e1bd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:08:24.550311 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-007b99f9-fdc7-4597-9a86-f8f888d3e1bd admin admin] REQUEST_METHOD: `DELETE` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:08:24.550764 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-007b99f9-fdc7-4597-9a86-f8f888d3e1bd admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:08:24.551218 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-007b99f9-fdc7-4597-9a86-f8f888d3e1bd admin admin] PATH_INFO: `/v3/users/275eec5bdb5340aba09304a01868a62e` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:08:24.553793 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-007b99f9-fdc7-4597-9a86-f8f888d3e1bd admin admin] RBAC: Authorizing `identity:delete_user(user_id=275eec5bdb5340aba09304a01868a62e)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:08:24.556285 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-007b99f9-fdc7-4597-9a86-f8f888d3e1bd admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:08:24.686965 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-007b99f9-fdc7-4597-9a86-f8f888d3e1bd admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '275eec5bdb5340aba09304a01868a62e'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:08:24.700548 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-007b99f9-fdc7-4597-9a86-f8f888d3e1bd admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '275eec5bdb5340aba09304a01868a62e'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:08:24.733688 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-007b99f9-fdc7-4597-9a86-f8f888d3e1bd admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '275eec5bdb5340aba09304a01868a62e'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:08:24.734409 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-007b99f9-fdc7-4597-9a86-f8f888d3e1bd admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '275eec5bdb5340aba09304a01868a62e'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:08:24.745877 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-007b99f9-fdc7-4597-9a86-f8f888d3e1bd admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '275eec5bdb5340aba09304a01868a62e'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:08:24.753855 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1004/2039] 10.222.0.50 () {58 vars in 1224 bytes} [Tue Sep 13 16:08:24 2022] DELETE /identity/v3/users/275eec5bdb5340aba09304a01868a62e => generated 0 bytes in 381 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 16:08:24.760238 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7169e57f-79b4-43ae-ad96-e415cb2a726f None None] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:08:24.760845 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7169e57f-79b4-43ae-ad96-e415cb2a726f None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:08:24.760988 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7169e57f-79b4-43ae-ad96-e415cb2a726f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:08:24.796521 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.auth.core [None req-7169e57f-79b4-43ae-ad96-e415cb2a726f None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81437) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:08:24.797970 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-7169e57f-79b4-43ae-ad96-e415cb2a726f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:08:24.860795 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-007b99f9-fdc7-4597-9a86-f8f888d3e1bd admin admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:08:24.863394 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-007b99f9-fdc7-4597-9a86-f8f888d3e1bd admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:08:24.917662 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1036/2040] 10.222.0.50 () {58 vars in 958 bytes} [Tue Sep 13 16:08:24 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 159 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:08:24.928736 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7169e57f-79b4-43ae-ad96-e415cb2a726f None None] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:08:24.930374 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-7169e57f-79b4-43ae-ad96-e415cb2a726f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:08:24.953874 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-007b99f9-fdc7-4597-9a86-f8f888d3e1bd admin admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:08:24.958761 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8a1e8cb-c57b-4008-936d-743c0c3a3094 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:08:24.959661 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8a1e8cb-c57b-4008-936d-743c0c3a3094 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:08:24.960011 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8a1e8cb-c57b-4008-936d-743c0c3a3094 admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:08:24.960487 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8a1e8cb-c57b-4008-936d-743c0c3a3094 admin admin] PATH_INFO: `/v3/projects/b19cf4202269486cae76f6df5b4a8635` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:08:24.961691 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8a1e8cb-c57b-4008-936d-743c0c3a3094 admin admin] RBAC: Authorizing `identity:delete_project(project_id=b19cf4202269486cae76f6df5b4a8635)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:08:24.963706 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8a1e8cb-c57b-4008-936d-743c0c3a3094 admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:08:24.990255 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7169e57f-79b4-43ae-ad96-e415cb2a726f None None] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:08:24.994590 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77467e9d-ec84-4fa5-b0ad-51463e7ef7a3 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f436f07939d4855a21b195bc7566244', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:08:24.995283 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77467e9d-ec84-4fa5-b0ad-51463e7ef7a3 service neutron] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:08:24.995584 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77467e9d-ec84-4fa5-b0ad-51463e7ef7a3 service neutron] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:08:24.995861 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77467e9d-ec84-4fa5-b0ad-51463e7ef7a3 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:08:25.005296 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77467e9d-ec84-4fa5-b0ad-51463e7ef7a3 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:08:25.007446 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77467e9d-ec84-4fa5-b0ad-51463e7ef7a3 service neutron] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:08:25.014146 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-f8a1e8cb-c57b-4008-936d-743c0c3a3094 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'b19cf4202269486cae76f6df5b4a8635', 'request_id': 'req-f8a1e8cb-c57b-4008-936d-743c0c3a3094'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:08:25.016836 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1037/2041] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 16:08:24 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 89 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 16:08:25.023178 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-f8a1e8cb-c57b-4008-936d-743c0c3a3094 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'b19cf4202269486cae76f6df5b4a8635', 'request_id': 'req-f8a1e8cb-c57b-4008-936d-743c0c3a3094'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:08:25.030596 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-f8a1e8cb-c57b-4008-936d-743c0c3a3094 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'b19cf4202269486cae76f6df5b4a8635', 'request_id': 'req-f8a1e8cb-c57b-4008-936d-743c0c3a3094'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:08:25.046361 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=81,keystone:DELETE=17,keystone:INSERT=2 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:08:25.052331 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-f8a1e8cb-c57b-4008-936d-743c0c3a3094 admin admin] The token cache is being invalidate because project b19cf4202269486cae76f6df5b4a8635 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81438) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 13 16:08:25.052985 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-f8a1e8cb-c57b-4008-936d-743c0c3a3094 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:08:25.054098 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-f8a1e8cb-c57b-4008-936d-743c0c3a3094 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:08:25.055937 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1005/2042] 10.222.0.50 () {58 vars in 1233 bytes} [Tue Sep 13 16:08:24 2022] DELETE /identity/v3/projects/b19cf4202269486cae76f6df5b4a8635 => generated 0 bytes in 196 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 13 16:08:25.060711 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31d25a51-76ec-46a4-9f6a-462addb51113 None None] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:08:25.060996 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31d25a51-76ec-46a4-9f6a-462addb51113 None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:08:25.061269 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31d25a51-76ec-46a4-9f6a-462addb51113 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:08:25.069364 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:08:25.084710 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.auth.core [None req-31d25a51-76ec-46a4-9f6a-462addb51113 None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81437) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:08:25.086143 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-31d25a51-76ec-46a4-9f6a-462addb51113 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:08:25.151509 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1038/2043] 10.222.0.50 () {58 vars in 958 bytes} [Tue Sep 13 16:08:25 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:08:25.159101 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e9daf38-3fb6-40ee-a9b8-f68e6fa7cff5 None None] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:08:25.159500 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e9daf38-3fb6-40ee-a9b8-f68e6fa7cff5 None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:08:25.159876 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e9daf38-3fb6-40ee-a9b8-f68e6fa7cff5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:08:25.181628 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.auth.core [None req-8e9daf38-3fb6-40ee-a9b8-f68e6fa7cff5 None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81438) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:08:25.183145 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-8e9daf38-3fb6-40ee-a9b8-f68e6fa7cff5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:08:25.191879 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1006/2044] 10.222.0.50 () {58 vars in 958 bytes} [Tue Sep 13 16:08:25 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:08:25.198130 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31d25a51-76ec-46a4-9f6a-462addb51113 None None] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:08:25.213077 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31d25a51-76ec-46a4-9f6a-462addb51113 None None] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:08:25.218157 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-967fa496-354c-4815-a4ef-ad104a723dbc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:08:25.218769 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-967fa496-354c-4815-a4ef-ad104a723dbc admin admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:08:25.219079 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-967fa496-354c-4815-a4ef-ad104a723dbc admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:08:25.219354 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-967fa496-354c-4815-a4ef-ad104a723dbc admin admin] PATH_INFO: `/v3/domains` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:08:25.220001 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-967fa496-354c-4815-a4ef-ad104a723dbc admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81437) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 16:08:25.220303 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-967fa496-354c-4815-a4ef-ad104a723dbc admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:08:25.221919 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-967fa496-354c-4815-a4ef-ad104a723dbc admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:08:25.235181 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1039/2045] 10.222.0.50 () {58 vars in 1153 bytes} [Tue Sep 13 16:08:25 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 13 16:08:25.240737 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e9daf38-3fb6-40ee-a9b8-f68e6fa7cff5 None None] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:08:25.250430 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e9daf38-3fb6-40ee-a9b8-f68e6fa7cff5 None None] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:08:25.256141 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e212549-4141-4061-b969-a9d73bdf5c3e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:08:25.256741 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e212549-4141-4061-b969-a9d73bdf5c3e admin admin] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:08:25.257021 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e212549-4141-4061-b969-a9d73bdf5c3e admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:08:25.257294 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e212549-4141-4061-b969-a9d73bdf5c3e admin admin] PATH_INFO: `/v3/projects` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:08:25.257929 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e212549-4141-4061-b969-a9d73bdf5c3e admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:08:25.259901 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e212549-4141-4061-b969-a9d73bdf5c3e admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:08:25.289301 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1007/2046] 10.222.0.50 () {60 vars in 1153 bytes} [Tue Sep 13 16:08:25 2022] POST /identity/v3/projects => generated 387 bytes in 50 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 16:08:25.293441 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-967fa496-354c-4815-a4ef-ad104a723dbc admin admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:08:25.304889 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-967fa496-354c-4815-a4ef-ad104a723dbc admin admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:08:25.315384 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e212549-4141-4061-b969-a9d73bdf5c3e admin admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:08:25.317053 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-1e212549-4141-4061-b969-a9d73bdf5c3e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:08:25.391413 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-990a9938-e006-4f0e-9326-bab39e0c6cb7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:08:25.392027 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-990a9938-e006-4f0e-9326-bab39e0c6cb7 admin admin] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:08:25.392307 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-990a9938-e006-4f0e-9326-bab39e0c6cb7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:08:25.392610 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-990a9938-e006-4f0e-9326-bab39e0c6cb7 admin admin] PATH_INFO: `/v3/users` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:08:25.393247 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-990a9938-e006-4f0e-9326-bab39e0c6cb7 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:08:25.395196 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-990a9938-e006-4f0e-9326-bab39e0c6cb7 admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:08:25.395796 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e212549-4141-4061-b969-a9d73bdf5c3e admin admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:08:25.400740 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ab4c920-ba66-46aa-a5b7-d4b8c87b6bb4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:08:25.401380 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ab4c920-ba66-46aa-a5b7-d4b8c87b6bb4 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:08:25.401659 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ab4c920-ba66-46aa-a5b7-d4b8c87b6bb4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:08:25.401931 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ab4c920-ba66-46aa-a5b7-d4b8c87b6bb4 admin admin] PATH_INFO: `/v3/projects/1dfc0424cf874568aeb9edd87d6905ec` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:08:25.403081 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ab4c920-ba66-46aa-a5b7-d4b8c87b6bb4 admin admin] RBAC: Authorizing `identity:delete_project(project_id=1dfc0424cf874568aeb9edd87d6905ec)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:08:25.405064 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ab4c920-ba66-46aa-a5b7-d4b8c87b6bb4 admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:08:25.425078 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1040/2047] 10.222.0.50 () {60 vars in 1144 bytes} [Tue Sep 13 16:08:25 2022] POST /identity/v3/users => generated 294 bytes in 129 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 16:08:25.427525 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-990a9938-e006-4f0e-9326-bab39e0c6cb7 admin admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:08:25.439660 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-990a9938-e006-4f0e-9326-bab39e0c6cb7 admin admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:08:25.446277 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7aafe8fc-5fb6-4fe9-b338-c650a5b3bc0f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:08:25.446939 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7aafe8fc-5fb6-4fe9-b338-c650a5b3bc0f admin admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:08:25.447223 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7aafe8fc-5fb6-4fe9-b338-c650a5b3bc0f admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:08:25.447564 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7aafe8fc-5fb6-4fe9-b338-c650a5b3bc0f admin admin] PATH_INFO: `/v3/roles` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:08:25.448173 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7aafe8fc-5fb6-4fe9-b338-c650a5b3bc0f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:08:25.449816 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7aafe8fc-5fb6-4fe9-b338-c650a5b3bc0f admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:08:25.464839 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1041/2048] 10.222.0.50 () {58 vars in 1122 bytes} [Tue Sep 13 16:08:25 2022] GET /identity/v3/roles => generated 1420 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 13 16:08:25.468537 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7aafe8fc-5fb6-4fe9-b338-c650a5b3bc0f admin admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:08:25.469290 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-2ab4c920-ba66-46aa-a5b7-d4b8c87b6bb4 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '1dfc0424cf874568aeb9edd87d6905ec', 'request_id': 'req-2ab4c920-ba66-46aa-a5b7-d4b8c87b6bb4'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:08:25.479933 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7aafe8fc-5fb6-4fe9-b338-c650a5b3bc0f admin admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:08:25.480421 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-2ab4c920-ba66-46aa-a5b7-d4b8c87b6bb4 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '1dfc0424cf874568aeb9edd87d6905ec', 'request_id': 'req-2ab4c920-ba66-46aa-a5b7-d4b8c87b6bb4'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:08:25.490706 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-2ab4c920-ba66-46aa-a5b7-d4b8c87b6bb4 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '1dfc0424cf874568aeb9edd87d6905ec', 'request_id': 'req-2ab4c920-ba66-46aa-a5b7-d4b8c87b6bb4'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:08:25.504962 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-2ab4c920-ba66-46aa-a5b7-d4b8c87b6bb4 admin admin] The token cache is being invalidate because project 1dfc0424cf874568aeb9edd87d6905ec was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81438) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 13 16:08:25.506123 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-2ab4c920-ba66-46aa-a5b7-d4b8c87b6bb4 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:08:25.507040 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-2ab4c920-ba66-46aa-a5b7-d4b8c87b6bb4 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:08:25.509199 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1008/2049] 10.222.0.50 () {58 vars in 1233 bytes} [Tue Sep 13 16:08:25 2022] DELETE /identity/v3/projects/1dfc0424cf874568aeb9edd87d6905ec => generated 0 bytes in 194 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 13 16:08:25.513228 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ab4c920-ba66-46aa-a5b7-d4b8c87b6bb4 admin admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:08:25.516159 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-2ab4c920-ba66-46aa-a5b7-d4b8c87b6bb4 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:08:25.585412 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7bb368e-2eb0-4c8b-aab0-83058c9f0156 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:08:25.586773 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7bb368e-2eb0-4c8b-aab0-83058c9f0156 admin admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:08:25.587087 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7bb368e-2eb0-4c8b-aab0-83058c9f0156 admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:08:25.587363 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7bb368e-2eb0-4c8b-aab0-83058c9f0156 admin admin] PATH_INFO: `/v3/roles` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:08:25.588698 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7bb368e-2eb0-4c8b-aab0-83058c9f0156 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:08:25.590317 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7bb368e-2eb0-4c8b-aab0-83058c9f0156 admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:08:25.602938 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1042/2050] 10.222.0.50 () {58 vars in 1122 bytes} [Tue Sep 13 16:08:25 2022] GET /identity/v3/roles => generated 1420 bytes in 134 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 16:08:25.607037 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7bb368e-2eb0-4c8b-aab0-83058c9f0156 admin admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:08:25.608834 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-b7bb368e-2eb0-4c8b-aab0-83058c9f0156 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:08:25.619211 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ab4c920-ba66-46aa-a5b7-d4b8c87b6bb4 admin admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:08:25.625293 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fea049e6-d8a1-45e8-be53-e093a91569ea admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:08:25.626038 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fea049e6-d8a1-45e8-be53-e093a91569ea admin admin] REQUEST_METHOD: `DELETE` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:08:25.626389 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fea049e6-d8a1-45e8-be53-e093a91569ea admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:08:25.626731 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fea049e6-d8a1-45e8-be53-e093a91569ea admin admin] PATH_INFO: `/v3/users/3ca4bc81953441a5b94d18556a1a9621` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:08:25.627995 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fea049e6-d8a1-45e8-be53-e093a91569ea admin admin] RBAC: Authorizing `identity:delete_user(user_id=3ca4bc81953441a5b94d18556a1a9621)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:08:25.629724 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:08:25.630261 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fea049e6-d8a1-45e8-be53-e093a91569ea admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:08:25.633549 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=91,keystone:INSERT=2,keystone:DELETE=7 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:08:25.635739 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7bb368e-2eb0-4c8b-aab0-83058c9f0156 admin admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:08:25.642319 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbfeb06d-6bf1-432a-a7fc-2cf21a4f69f9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:08:25.643085 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbfeb06d-6bf1-432a-a7fc-2cf21a4f69f9 admin admin] REQUEST_METHOD: `PUT` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:08:25.643364 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbfeb06d-6bf1-432a-a7fc-2cf21a4f69f9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:08:25.643636 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbfeb06d-6bf1-432a-a7fc-2cf21a4f69f9 admin admin] PATH_INFO: `/v3/projects/ab1ec172a97348899465eaeba821aad9/users/37be89f5eb8f4715a0a1f7ea1a3ccae6/roles/15db4aaca3264ae29bde38903ba43471` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:08:25.664139 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbfeb06d-6bf1-432a-a7fc-2cf21a4f69f9 admin admin] RBAC: Authorizing `identity:create_grant(project_id=ab1ec172a97348899465eaeba821aad9, user_id=37be89f5eb8f4715a0a1f7ea1a3ccae6, role_id=15db4aaca3264ae29bde38903ba43471)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:08:25.667000 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbfeb06d-6bf1-432a-a7fc-2cf21a4f69f9 admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:08:25.689908 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1043/2051] 10.222.0.50 () {60 vars in 1483 bytes} [Tue Sep 13 16:08:25 2022] PUT /identity/v3/projects/ab1ec172a97348899465eaeba821aad9/users/37be89f5eb8f4715a0a1f7ea1a3ccae6/roles/15db4aaca3264ae29bde38903ba43471 => generated 0 bytes in 84 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 13 16:08:25.695540 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-fea049e6-d8a1-45e8-be53-e093a91569ea admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '3ca4bc81953441a5b94d18556a1a9621'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:08:25.700388 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1105f667-53c5-41c4-8d07-97bf7574aefe None None] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:08:25.700836 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1105f667-53c5-41c4-8d07-97bf7574aefe None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:08:25.700970 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1105f667-53c5-41c4-8d07-97bf7574aefe None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:08:25.704548 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-fea049e6-d8a1-45e8-be53-e093a91569ea admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '3ca4bc81953441a5b94d18556a1a9621'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:08:25.720560 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-fea049e6-d8a1-45e8-be53-e093a91569ea admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '3ca4bc81953441a5b94d18556a1a9621'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:08:25.721287 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-fea049e6-d8a1-45e8-be53-e093a91569ea admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '3ca4bc81953441a5b94d18556a1a9621'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:08:25.730423 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-fea049e6-d8a1-45e8-be53-e093a91569ea admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '3ca4bc81953441a5b94d18556a1a9621'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:08:25.730930 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.auth.core [None req-1105f667-53c5-41c4-8d07-97bf7574aefe None None] MFA Rules not processed for user `37be89f5eb8f4715a0a1f7ea1a3ccae6`. Rule list: `[]` (Enabled: `True`). {{(pid=81437) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:08:25.732296 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-1105f667-53c5-41c4-8d07-97bf7574aefe None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:08:25.741731 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1009/2052] 10.222.0.50 () {58 vars in 1224 bytes} [Tue Sep 13 16:08:25 2022] DELETE /identity/v3/users/3ca4bc81953441a5b94d18556a1a9621 => generated 0 bytes in 229 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 16:08:25.837627 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1044/2053] 10.222.0.50 () {58 vars in 958 bytes} [Tue Sep 13 16:08:25 2022] POST /identity/v3/auth/tokens => generated 3264 bytes in 138 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 13 16:08:25.842986 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fea049e6-d8a1-45e8-be53-e093a91569ea admin admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:08:25.845054 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-fea049e6-d8a1-45e8-be53-e093a91569ea admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:08:25.947446 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fea049e6-d8a1-45e8-be53-e093a91569ea admin admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:08:25.952447 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09389ec5-a74f-48b8-8106-3835ea5b1902 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:08:25.953103 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09389ec5-a74f-48b8-8106-3835ea5b1902 admin admin] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:08:25.953381 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09389ec5-a74f-48b8-8106-3835ea5b1902 admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:08:25.953651 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09389ec5-a74f-48b8-8106-3835ea5b1902 admin admin] PATH_INFO: `/v3/projects` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:08:25.954281 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09389ec5-a74f-48b8-8106-3835ea5b1902 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:08:25.956215 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09389ec5-a74f-48b8-8106-3835ea5b1902 admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:08:26.009999 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1010/2054] 10.222.0.50 () {60 vars in 1153 bytes} [Tue Sep 13 16:08:25 2022] POST /identity/v3/projects => generated 387 bytes in 168 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 16:08:26.015256 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1105f667-53c5-41c4-8d07-97bf7574aefe None None] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:08:26.089570 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1105f667-53c5-41c4-8d07-97bf7574aefe None None] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:08:26.166406 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e7daeda-1008-46bd-bbc3-69128d5b28e7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:08:26.174068 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e7daeda-1008-46bd-bbc3-69128d5b28e7 admin admin] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:08:26.174068 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e7daeda-1008-46bd-bbc3-69128d5b28e7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:08:26.174068 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e7daeda-1008-46bd-bbc3-69128d5b28e7 admin admin] PATH_INFO: `/v3/users` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:08:26.174068 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e7daeda-1008-46bd-bbc3-69128d5b28e7 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:08:26.174489 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09389ec5-a74f-48b8-8106-3835ea5b1902 admin admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:08:26.174593 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e7daeda-1008-46bd-bbc3-69128d5b28e7 admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:08:26.174682 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-09389ec5-a74f-48b8-8106-3835ea5b1902 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:08:26.189539 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09389ec5-a74f-48b8-8106-3835ea5b1902 admin admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:08:26.196722 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1045/2055] 10.222.0.50 () {60 vars in 1144 bytes} [Tue Sep 13 16:08:26 2022] POST /identity/v3/users => generated 300 bytes in 182 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 16:08:26.201125 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e7daeda-1008-46bd-bbc3-69128d5b28e7 admin admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:08:26.206924 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa443530-efb6-45b2-a845-a90528d7d71b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:08:26.209919 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa443530-efb6-45b2-a845-a90528d7d71b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:08:26.210616 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa443530-efb6-45b2-a845-a90528d7d71b admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:08:26.211215 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa443530-efb6-45b2-a845-a90528d7d71b admin admin] PATH_INFO: `/v3/projects/c8fc3b46ad7b44108a730a9c1dfe6b6e` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:08:26.213456 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e7daeda-1008-46bd-bbc3-69128d5b28e7 admin admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:08:26.213838 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa443530-efb6-45b2-a845-a90528d7d71b admin admin] RBAC: Authorizing `identity:delete_project(project_id=c8fc3b46ad7b44108a730a9c1dfe6b6e)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:08:26.216054 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa443530-efb6-45b2-a845-a90528d7d71b admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:08:26.218694 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8249565a-2a7e-419e-a955-e8778ad8a1f6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:08:26.219419 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8249565a-2a7e-419e-a955-e8778ad8a1f6 admin admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:08:26.219766 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8249565a-2a7e-419e-a955-e8778ad8a1f6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:08:26.220100 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8249565a-2a7e-419e-a955-e8778ad8a1f6 admin admin] PATH_INFO: `/v3/roles` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:08:26.220817 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8249565a-2a7e-419e-a955-e8778ad8a1f6 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:08:26.222471 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8249565a-2a7e-419e-a955-e8778ad8a1f6 admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:08:26.224831 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:08:26.237884 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1046/2056] 10.222.0.50 () {58 vars in 1122 bytes} [Tue Sep 13 16:08:26 2022] GET /identity/v3/roles => generated 1420 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 13 16:08:26.241726 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8249565a-2a7e-419e-a955-e8778ad8a1f6 admin admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:08:26.254367 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8249565a-2a7e-419e-a955-e8778ad8a1f6 admin admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:08:26.261519 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b8117c1-1445-4187-b972-a517033c0ebd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:08:26.262288 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b8117c1-1445-4187-b972-a517033c0ebd admin admin] REQUEST_METHOD: `PUT` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:08:26.262648 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b8117c1-1445-4187-b972-a517033c0ebd admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:08:26.262952 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b8117c1-1445-4187-b972-a517033c0ebd admin admin] PATH_INFO: `/v3/projects/f1ef7af8595e401abaf4564a5e6d2085/users/53f033391a5d46d0a973b8ddb0681b5e/roles/031d4e1a23e34f71858fc4a6dfbd4ccb` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:08:26.274635 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-fa443530-efb6-45b2-a845-a90528d7d71b admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'c8fc3b46ad7b44108a730a9c1dfe6b6e', 'request_id': 'req-fa443530-efb6-45b2-a845-a90528d7d71b'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:08:26.280832 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b8117c1-1445-4187-b972-a517033c0ebd admin admin] RBAC: Authorizing `identity:create_grant(project_id=f1ef7af8595e401abaf4564a5e6d2085, user_id=53f033391a5d46d0a973b8ddb0681b5e, role_id=031d4e1a23e34f71858fc4a6dfbd4ccb)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:08:26.283563 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b8117c1-1445-4187-b972-a517033c0ebd admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:08:26.283928 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-fa443530-efb6-45b2-a845-a90528d7d71b admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'c8fc3b46ad7b44108a730a9c1dfe6b6e', 'request_id': 'req-fa443530-efb6-45b2-a845-a90528d7d71b'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:08:26.289332 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=81,keystone:DELETE=16,keystone:INSERT=3 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:08:26.292156 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-fa443530-efb6-45b2-a845-a90528d7d71b admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'c8fc3b46ad7b44108a730a9c1dfe6b6e', 'request_id': 'req-fa443530-efb6-45b2-a845-a90528d7d71b'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:08:26.299307 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1047/2057] 10.222.0.50 () {60 vars in 1483 bytes} [Tue Sep 13 16:08:26 2022] PUT /identity/v3/projects/f1ef7af8595e401abaf4564a5e6d2085/users/53f033391a5d46d0a973b8ddb0681b5e/roles/031d4e1a23e34f71858fc4a6dfbd4ccb => generated 0 bytes in 58 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 13 16:08:26.303751 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b8117c1-1445-4187-b972-a517033c0ebd admin admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:08:26.313137 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-fa443530-efb6-45b2-a845-a90528d7d71b admin admin] The token cache is being invalidate because project c8fc3b46ad7b44108a730a9c1dfe6b6e was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81438) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 13 16:08:26.313567 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-fa443530-efb6-45b2-a845-a90528d7d71b admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:08:26.314474 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-fa443530-efb6-45b2-a845-a90528d7d71b admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:08:26.330121 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1011/2058] 10.222.0.50 () {58 vars in 1233 bytes} [Tue Sep 13 16:08:26 2022] DELETE /identity/v3/projects/c8fc3b46ad7b44108a730a9c1dfe6b6e => generated 0 bytes in 161 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 16:08:26.336707 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7257b4dc-3235-4357-a21a-1f3c22ae63dc None None] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:08:26.336707 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7257b4dc-3235-4357-a21a-1f3c22ae63dc None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:08:26.337000 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7257b4dc-3235-4357-a21a-1f3c22ae63dc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:08:26.360058 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.auth.core [None req-7257b4dc-3235-4357-a21a-1f3c22ae63dc None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81438) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:08:26.361425 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-7257b4dc-3235-4357-a21a-1f3c22ae63dc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:08:26.409231 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b8117c1-1445-4187-b972-a517033c0ebd admin admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:08:26.415204 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45886d20-5b07-48bb-b3ba-4bde7f93b507 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:08:26.415828 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45886d20-5b07-48bb-b3ba-4bde7f93b507 admin admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:08:26.416114 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45886d20-5b07-48bb-b3ba-4bde7f93b507 admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:08:26.416403 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45886d20-5b07-48bb-b3ba-4bde7f93b507 admin admin] PATH_INFO: `/v3/roles` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:08:26.417261 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45886d20-5b07-48bb-b3ba-4bde7f93b507 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:08:26.418892 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45886d20-5b07-48bb-b3ba-4bde7f93b507 admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:08:26.429625 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1048/2059] 10.222.0.50 () {58 vars in 1122 bytes} [Tue Sep 13 16:08:26 2022] GET /identity/v3/roles => generated 1420 bytes in 126 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 13 16:08:26.434029 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45886d20-5b07-48bb-b3ba-4bde7f93b507 admin admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:08:26.435995 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-45886d20-5b07-48bb-b3ba-4bde7f93b507 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:08:26.439457 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1012/2060] 10.222.0.50 () {58 vars in 958 bytes} [Tue Sep 13 16:08:26 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 13 16:08:26.448122 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a984218-e493-4125-bb9e-9c077a793003 None None] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:08:26.448416 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a984218-e493-4125-bb9e-9c077a793003 None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:08:26.448721 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a984218-e493-4125-bb9e-9c077a793003 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:08:26.455718 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45886d20-5b07-48bb-b3ba-4bde7f93b507 admin admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:08:26.461468 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fec06cf-1185-457f-b33b-226b413d8e4f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:08:26.465104 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fec06cf-1185-457f-b33b-226b413d8e4f admin admin] REQUEST_METHOD: `PUT` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:08:26.465639 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fec06cf-1185-457f-b33b-226b413d8e4f admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:08:26.466091 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fec06cf-1185-457f-b33b-226b413d8e4f admin admin] PATH_INFO: `/v3/projects/f1ef7af8595e401abaf4564a5e6d2085/users/53f033391a5d46d0a973b8ddb0681b5e/roles/031d4e1a23e34f71858fc4a6dfbd4ccb` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:08:26.468455 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fec06cf-1185-457f-b33b-226b413d8e4f admin admin] RBAC: Authorizing `identity:create_grant(project_id=f1ef7af8595e401abaf4564a5e6d2085, user_id=53f033391a5d46d0a973b8ddb0681b5e, role_id=031d4e1a23e34f71858fc4a6dfbd4ccb)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:08:26.472646 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fec06cf-1185-457f-b33b-226b413d8e4f admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:08:26.474440 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.auth.core [None req-3a984218-e493-4125-bb9e-9c077a793003 None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81438) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:08:26.475337 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-3a984218-e493-4125-bb9e-9c077a793003 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:08:26.484576 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1013/2061] 10.222.0.50 () {58 vars in 958 bytes} [Tue Sep 13 16:08:26 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 37 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 13 16:08:26.488875 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a984218-e493-4125-bb9e-9c077a793003 None None] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:08:26.501753 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a984218-e493-4125-bb9e-9c077a793003 None None] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:08:26.503237 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1049/2062] 10.222.0.50 () {60 vars in 1483 bytes} [Tue Sep 13 16:08:26 2022] PUT /identity/v3/projects/f1ef7af8595e401abaf4564a5e6d2085/users/53f033391a5d46d0a973b8ddb0681b5e/roles/031d4e1a23e34f71858fc4a6dfbd4ccb => generated 0 bytes in 59 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 13 16:08:26.505351 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5f021e0-ecf2-43b9-847b-94c7f5df3447 None None] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:08:26.505893 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5f021e0-ecf2-43b9-847b-94c7f5df3447 None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:08:26.506207 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5f021e0-ecf2-43b9-847b-94c7f5df3447 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:08:26.536538 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.auth.core [None req-d5f021e0-ecf2-43b9-847b-94c7f5df3447 None None] MFA Rules not processed for user `53f033391a5d46d0a973b8ddb0681b5e`. Rule list: `[]` (Enabled: `True`). {{(pid=81437) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:08:26.537672 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-d5f021e0-ecf2-43b9-847b-94c7f5df3447 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:08:26.598818 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35605ec8-495f-40c9-af24-86434d144a1e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:08:26.599582 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35605ec8-495f-40c9-af24-86434d144a1e admin admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:08:26.599933 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35605ec8-495f-40c9-af24-86434d144a1e admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:08:26.600282 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35605ec8-495f-40c9-af24-86434d144a1e admin admin] PATH_INFO: `/v3/domains` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:08:26.601394 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35605ec8-495f-40c9-af24-86434d144a1e admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81438) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 16:08:26.601394 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35605ec8-495f-40c9-af24-86434d144a1e admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:08:26.602999 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35605ec8-495f-40c9-af24-86434d144a1e admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:08:26.615305 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1014/2063] 10.222.0.50 () {58 vars in 1153 bytes} [Tue Sep 13 16:08:26 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 127 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 16:08:26.623272 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35605ec8-495f-40c9-af24-86434d144a1e admin admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:08:26.630347 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1050/2064] 10.222.0.50 () {58 vars in 958 bytes} [Tue Sep 13 16:08:26 2022] POST /identity/v3/auth/tokens => generated 3331 bytes in 127 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:08:26.639976 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35605ec8-495f-40c9-af24-86434d144a1e admin admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:08:26.642201 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5f021e0-ecf2-43b9-847b-94c7f5df3447 None None] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:08:26.643729 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-d5f021e0-ecf2-43b9-847b-94c7f5df3447 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:08:26.645232 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43c16e23-4e9f-4204-9ca9-48b7c7870323 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:08:26.646741 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43c16e23-4e9f-4204-9ca9-48b7c7870323 admin admin] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:08:26.647121 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43c16e23-4e9f-4204-9ca9-48b7c7870323 admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:08:26.647502 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43c16e23-4e9f-4204-9ca9-48b7c7870323 admin admin] PATH_INFO: `/v3/projects` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:08:26.648159 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43c16e23-4e9f-4204-9ca9-48b7c7870323 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:08:26.650265 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43c16e23-4e9f-4204-9ca9-48b7c7870323 admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:08:26.680931 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1015/2065] 10.222.0.50 () {60 vars in 1153 bytes} [Tue Sep 13 16:08:26 2022] POST /identity/v3/projects => generated 377 bytes in 58 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 16:08:26.685230 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43c16e23-4e9f-4204-9ca9-48b7c7870323 admin admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:08:26.695486 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:08:26.713366 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5f021e0-ecf2-43b9-847b-94c7f5df3447 None None] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:08:26.774564 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88f15f77-5ea1-4e6a-a2de-805c459d6904 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7fd6b5afd67b48f9b0f0597ccd4cf4df', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:08:26.775354 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88f15f77-5ea1-4e6a-a2de-805c459d6904 service cinder] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:08:26.775723 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88f15f77-5ea1-4e6a-a2de-805c459d6904 service cinder] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:08:26.776071 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88f15f77-5ea1-4e6a-a2de-805c459d6904 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:08:26.778470 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:DELETE=1,keystone:SELECT=98,keystone:INSERT=1 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:08:26.785374 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43c16e23-4e9f-4204-9ca9-48b7c7870323 admin admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:08:26.791906 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-544ee3df-93ab-4fdf-a239-646df739fedb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:08:26.792483 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-544ee3df-93ab-4fdf-a239-646df739fedb admin admin] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:08:26.792763 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-544ee3df-93ab-4fdf-a239-646df739fedb admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:08:26.793037 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-544ee3df-93ab-4fdf-a239-646df739fedb admin admin] PATH_INFO: `/v3/users` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:08:26.793670 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-544ee3df-93ab-4fdf-a239-646df739fedb admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:08:26.795637 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-544ee3df-93ab-4fdf-a239-646df739fedb admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:08:26.820546 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1016/2066] 10.222.0.50 () {60 vars in 1144 bytes} [Tue Sep 13 16:08:26 2022] POST /identity/v3/users => generated 289 bytes in 132 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 13 16:08:26.821603 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-544ee3df-93ab-4fdf-a239-646df739fedb admin admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:08:26.832000 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-544ee3df-93ab-4fdf-a239-646df739fedb admin admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:08:26.837911 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b01038de-c8a4-4522-b2e3-d9984bc653d4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:08:26.838589 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b01038de-c8a4-4522-b2e3-d9984bc653d4 admin admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:08:26.838966 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b01038de-c8a4-4522-b2e3-d9984bc653d4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:08:26.839306 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b01038de-c8a4-4522-b2e3-d9984bc653d4 admin admin] PATH_INFO: `/v3/roles` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:08:26.840041 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b01038de-c8a4-4522-b2e3-d9984bc653d4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:08:26.843540 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b01038de-c8a4-4522-b2e3-d9984bc653d4 admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:08:26.846411 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88f15f77-5ea1-4e6a-a2de-805c459d6904 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:08:26.848596 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88f15f77-5ea1-4e6a-a2de-805c459d6904 service cinder] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:08:26.852976 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1017/2067] 10.222.0.50 () {58 vars in 1122 bytes} [Tue Sep 13 16:08:26 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 16:08:26.856839 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b01038de-c8a4-4522-b2e3-d9984bc653d4 admin admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:08:26.862390 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1051/2068] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 16:08:26 2022] GET /identity/v3/auth/tokens => generated 3331 bytes in 221 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 16:08:26.868655 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b01038de-c8a4-4522-b2e3-d9984bc653d4 admin admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:08:26.873307 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3873749-2b58-40a0-93d5-07e0e5cb2b51 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:08:26.873912 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3873749-2b58-40a0-93d5-07e0e5cb2b51 admin admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:08:26.874218 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3873749-2b58-40a0-93d5-07e0e5cb2b51 admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:08:26.874491 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3873749-2b58-40a0-93d5-07e0e5cb2b51 admin admin] PATH_INFO: `/v3/roles` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:08:26.875214 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3873749-2b58-40a0-93d5-07e0e5cb2b51 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:08:26.876934 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3873749-2b58-40a0-93d5-07e0e5cb2b51 admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:08:26.893458 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1018/2069] 10.222.0.50 () {58 vars in 1122 bytes} [Tue Sep 13 16:08:26 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 13 16:08:26.894648 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88f15f77-5ea1-4e6a-a2de-805c459d6904 service cinder] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:08:26.922213 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88f15f77-5ea1-4e6a-a2de-805c459d6904 service cinder] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:08:26.931247 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e6f3007-fc2b-4ceb-9e10-2c7f5f9f6e7c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:08:26.931929 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e6f3007-fc2b-4ceb-9e10-2c7f5f9f6e7c admin admin] REQUEST_METHOD: `PUT` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:08:26.932212 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e6f3007-fc2b-4ceb-9e10-2c7f5f9f6e7c admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:08:26.932488 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e6f3007-fc2b-4ceb-9e10-2c7f5f9f6e7c admin admin] PATH_INFO: `/v3/projects/d1b4f10d46fa43a7bd0619c3839ca8da/users/ea071cd9cef94e2e94cc4b63f4488680/roles/15db4aaca3264ae29bde38903ba43471` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:08:26.956591 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e6f3007-fc2b-4ceb-9e10-2c7f5f9f6e7c admin admin] RBAC: Authorizing `identity:create_grant(project_id=d1b4f10d46fa43a7bd0619c3839ca8da, user_id=ea071cd9cef94e2e94cc4b63f4488680, role_id=15db4aaca3264ae29bde38903ba43471)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:08:26.959696 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e6f3007-fc2b-4ceb-9e10-2c7f5f9f6e7c admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:08:26.976528 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1052/2070] 10.222.0.50 () {60 vars in 1483 bytes} [Tue Sep 13 16:08:26 2022] PUT /identity/v3/projects/d1b4f10d46fa43a7bd0619c3839ca8da/users/ea071cd9cef94e2e94cc4b63f4488680/roles/15db4aaca3264ae29bde38903ba43471 => generated 0 bytes in 82 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 16:08:26.987486 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52115be3-4dd7-461a-a639-9f691a076329 None None] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:08:26.987773 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52115be3-4dd7-461a-a639-9f691a076329 None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:08:26.988048 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52115be3-4dd7-461a-a639-9f691a076329 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:08:27.019720 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.auth.core [None req-52115be3-4dd7-461a-a639-9f691a076329 None None] MFA Rules not processed for user `ea071cd9cef94e2e94cc4b63f4488680`. Rule list: `[]` (Enabled: `True`). {{(pid=81438) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:08:27.021973 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-52115be3-4dd7-461a-a639-9f691a076329 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:08:27.098143 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1019/2071] 10.222.0.50 () {58 vars in 958 bytes} [Tue Sep 13 16:08:26 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 112 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:08:27.103277 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e6f3007-fc2b-4ceb-9e10-2c7f5f9f6e7c admin admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:08:27.170353 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e6f3007-fc2b-4ceb-9e10-2c7f5f9f6e7c admin admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:08:27.175419 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8414a376-635d-49ae-bcb3-342984174837 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:08:27.176199 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8414a376-635d-49ae-bcb3-342984174837 admin admin] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:08:27.176718 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8414a376-635d-49ae-bcb3-342984174837 admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:08:27.179415 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8414a376-635d-49ae-bcb3-342984174837 admin admin] PATH_INFO: `/v3/projects` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:08:27.184685 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8414a376-635d-49ae-bcb3-342984174837 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:08:27.186690 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8414a376-635d-49ae-bcb3-342984174837 admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:08:27.218648 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1053/2072] 10.222.0.50 () {60 vars in 1153 bytes} [Tue Sep 13 16:08:27 2022] POST /identity/v3/projects => generated 379 bytes in 116 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 16:08:27.223116 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52115be3-4dd7-461a-a639-9f691a076329 None None] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:08:27.236633 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52115be3-4dd7-461a-a639-9f691a076329 None None] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:08:27.296016 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11a8311d-90f0-4d5a-8444-77aedf46667e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:08:27.296630 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11a8311d-90f0-4d5a-8444-77aedf46667e admin admin] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:08:27.296912 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11a8311d-90f0-4d5a-8444-77aedf46667e admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:08:27.297184 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11a8311d-90f0-4d5a-8444-77aedf46667e admin admin] PATH_INFO: `/v3/users` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:08:27.297819 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11a8311d-90f0-4d5a-8444-77aedf46667e admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:08:27.299763 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11a8311d-90f0-4d5a-8444-77aedf46667e admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:08:27.322930 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1020/2073] 10.222.0.50 () {60 vars in 1144 bytes} [Tue Sep 13 16:08:27 2022] POST /identity/v3/users => generated 290 bytes in 101 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 13 16:08:27.326805 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8414a376-635d-49ae-bcb3-342984174837 admin admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:08:27.337969 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8414a376-635d-49ae-bcb3-342984174837 admin admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:08:27.343638 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c58a8824-2140-41ee-a2b6-33b6e2a79a9a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:08:27.344081 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c58a8824-2140-41ee-a2b6-33b6e2a79a9a admin admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:08:27.344376 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c58a8824-2140-41ee-a2b6-33b6e2a79a9a admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:08:27.344640 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c58a8824-2140-41ee-a2b6-33b6e2a79a9a admin admin] PATH_INFO: `/v3/roles` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:08:27.345297 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c58a8824-2140-41ee-a2b6-33b6e2a79a9a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:08:27.346933 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c58a8824-2140-41ee-a2b6-33b6e2a79a9a admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:08:27.349592 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:08:27.364874 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1054/2074] 10.222.0.50 () {58 vars in 1122 bytes} [Tue Sep 13 16:08:27 2022] GET /identity/v3/roles => generated 1420 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 16:08:27.368971 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11a8311d-90f0-4d5a-8444-77aedf46667e admin admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:08:27.378950 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11a8311d-90f0-4d5a-8444-77aedf46667e admin admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:08:27.384624 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5ff1766-9963-4251-8467-7226a6eda34e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:08:27.385721 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5ff1766-9963-4251-8467-7226a6eda34e admin admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:08:27.386033 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5ff1766-9963-4251-8467-7226a6eda34e admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:08:27.386391 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5ff1766-9963-4251-8467-7226a6eda34e admin admin] PATH_INFO: `/v3/roles` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:08:27.387342 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5ff1766-9963-4251-8467-7226a6eda34e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:08:27.389421 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5ff1766-9963-4251-8467-7226a6eda34e admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:08:27.402833 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1021/2075] 10.222.0.50 () {58 vars in 1122 bytes} [Tue Sep 13 16:08:27 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 13 16:08:27.406765 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c58a8824-2140-41ee-a2b6-33b6e2a79a9a admin admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:08:27.417847 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c58a8824-2140-41ee-a2b6-33b6e2a79a9a admin admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:08:27.422583 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-681e0f30-a265-4774-b3ec-9710a13c2b7e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:08:27.423290 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-681e0f30-a265-4774-b3ec-9710a13c2b7e admin admin] REQUEST_METHOD: `PUT` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:08:27.423571 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-681e0f30-a265-4774-b3ec-9710a13c2b7e admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:08:27.423849 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-681e0f30-a265-4774-b3ec-9710a13c2b7e admin admin] PATH_INFO: `/v3/projects/2ca361e398514a9f919ae53bd82a60ac/users/6978e9d2bd45491db3aaa7fd58e6f250/roles/15db4aaca3264ae29bde38903ba43471` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:08:27.447342 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-681e0f30-a265-4774-b3ec-9710a13c2b7e admin admin] RBAC: Authorizing `identity:create_grant(project_id=2ca361e398514a9f919ae53bd82a60ac, user_id=6978e9d2bd45491db3aaa7fd58e6f250, role_id=15db4aaca3264ae29bde38903ba43471)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:08:27.450160 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-681e0f30-a265-4774-b3ec-9710a13c2b7e admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:08:27.464183 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1055/2076] 10.222.0.50 () {60 vars in 1483 bytes} [Tue Sep 13 16:08:27 2022] PUT /identity/v3/projects/2ca361e398514a9f919ae53bd82a60ac/users/6978e9d2bd45491db3aaa7fd58e6f250/roles/15db4aaca3264ae29bde38903ba43471 => generated 0 bytes in 58 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 13 16:08:27.474796 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9003b2fb-224c-4a39-acff-74ec0c633c73 None None] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:08:27.475186 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9003b2fb-224c-4a39-acff-74ec0c633c73 None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:08:27.475471 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9003b2fb-224c-4a39-acff-74ec0c633c73 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:08:27.513681 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.auth.core [None req-9003b2fb-224c-4a39-acff-74ec0c633c73 None None] MFA Rules not processed for user `6978e9d2bd45491db3aaa7fd58e6f250`. Rule list: `[]` (Enabled: `True`). {{(pid=81438) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:08:27.517866 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-9003b2fb-224c-4a39-acff-74ec0c633c73 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:08:27.518333 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-681e0f30-a265-4774-b3ec-9710a13c2b7e admin admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:08:27.536897 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=94,keystone:INSERT=6 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:08:27.583059 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-681e0f30-a265-4774-b3ec-9710a13c2b7e admin admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:08:27.587872 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74869c53-4399-4562-be64-5e880162e1d8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:08:27.588603 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74869c53-4399-4562-be64-5e880162e1d8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:08:27.588810 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74869c53-4399-4562-be64-5e880162e1d8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:08:27.589049 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74869c53-4399-4562-be64-5e880162e1d8 admin admin] PATH_INFO: `/v3/users/37be89f5eb8f4715a0a1f7ea1a3ccae6` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:08:27.589982 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1022/2077] 10.222.0.50 () {58 vars in 958 bytes} [Tue Sep 13 16:08:27 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 116 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:08:27.594061 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74869c53-4399-4562-be64-5e880162e1d8 admin admin] RBAC: Authorizing `identity:delete_user(user_id=37be89f5eb8f4715a0a1f7ea1a3ccae6)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:08:27.594061 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74869c53-4399-4562-be64-5e880162e1d8 admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:08:27.594420 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9003b2fb-224c-4a39-acff-74ec0c633c73 None None] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:08:27.611561 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9003b2fb-224c-4a39-acff-74ec0c633c73 None None] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:08:27.618466 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76b52350-5e60-4e45-be65-7f6e04d6c915 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:08:27.619431 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76b52350-5e60-4e45-be65-7f6e04d6c915 admin admin] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:08:27.619431 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76b52350-5e60-4e45-be65-7f6e04d6c915 admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:08:27.619716 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76b52350-5e60-4e45-be65-7f6e04d6c915 admin admin] PATH_INFO: `/v3/projects` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:08:27.620492 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76b52350-5e60-4e45-be65-7f6e04d6c915 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:08:27.622228 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76b52350-5e60-4e45-be65-7f6e04d6c915 admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:08:27.655127 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-74869c53-4399-4562-be64-5e880162e1d8 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '37be89f5eb8f4715a0a1f7ea1a3ccae6'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:08:27.658051 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1023/2078] 10.222.0.50 () {60 vars in 1153 bytes} [Tue Sep 13 16:08:27 2022] POST /identity/v3/projects => generated 377 bytes in 64 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 13 16:08:27.662274 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76b52350-5e60-4e45-be65-7f6e04d6c915 admin admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:08:27.666001 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-74869c53-4399-4562-be64-5e880162e1d8 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '37be89f5eb8f4715a0a1f7ea1a3ccae6'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:08:27.684790 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-74869c53-4399-4562-be64-5e880162e1d8 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '37be89f5eb8f4715a0a1f7ea1a3ccae6'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:08:27.685565 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-74869c53-4399-4562-be64-5e880162e1d8 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '37be89f5eb8f4715a0a1f7ea1a3ccae6'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:08:27.694730 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-74869c53-4399-4562-be64-5e880162e1d8 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '37be89f5eb8f4715a0a1f7ea1a3ccae6'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:08:27.704910 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1056/2079] 10.222.0.50 () {58 vars in 1224 bytes} [Tue Sep 13 16:08:27 2022] DELETE /identity/v3/users/37be89f5eb8f4715a0a1f7ea1a3ccae6 => generated 0 bytes in 184 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 13 16:08:27.706370 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb203373-b2cc-4387-8e4e-b3e064534141 None None] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:08:27.706721 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb203373-b2cc-4387-8e4e-b3e064534141 None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:08:27.707088 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb203373-b2cc-4387-8e4e-b3e064534141 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:08:27.729127 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.auth.core [None req-eb203373-b2cc-4387-8e4e-b3e064534141 None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81437) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:08:27.730446 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-eb203373-b2cc-4387-8e4e-b3e064534141 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:08:27.741916 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76b52350-5e60-4e45-be65-7f6e04d6c915 admin admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:08:27.780839 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=88,keystone:INSERT=2,keystone:DELETE=10 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:08:27.803877 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1057/2080] 10.222.0.50 () {58 vars in 958 bytes} [Tue Sep 13 16:08:27 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 13 16:08:27.815950 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb203373-b2cc-4387-8e4e-b3e064534141 None None] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:08:27.818498 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-eb203373-b2cc-4387-8e4e-b3e064534141 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:08:27.822636 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5830e762-7b35-45a1-b408-47537fb12d96 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:08:27.823702 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5830e762-7b35-45a1-b408-47537fb12d96 admin admin] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:08:27.826115 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5830e762-7b35-45a1-b408-47537fb12d96 admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:08:27.826563 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5830e762-7b35-45a1-b408-47537fb12d96 admin admin] PATH_INFO: `/v3/users` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:08:27.827299 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5830e762-7b35-45a1-b408-47537fb12d96 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:08:27.829236 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5830e762-7b35-45a1-b408-47537fb12d96 admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:08:27.855907 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1024/2081] 10.222.0.50 () {60 vars in 1144 bytes} [Tue Sep 13 16:08:27 2022] POST /identity/v3/users => generated 295 bytes in 194 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 13 16:08:27.859798 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5830e762-7b35-45a1-b408-47537fb12d96 admin admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:08:27.862357 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-5830e762-7b35-45a1-b408-47537fb12d96 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:08:27.875962 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5830e762-7b35-45a1-b408-47537fb12d96 admin admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:08:27.882109 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e38f67f6-0838-49d3-a4db-d22bbcd608d8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:08:27.882723 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e38f67f6-0838-49d3-a4db-d22bbcd608d8 admin admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:08:27.883310 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e38f67f6-0838-49d3-a4db-d22bbcd608d8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:08:27.883591 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e38f67f6-0838-49d3-a4db-d22bbcd608d8 admin admin] PATH_INFO: `/v3/roles` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:08:27.884238 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e38f67f6-0838-49d3-a4db-d22bbcd608d8 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:08:27.885949 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e38f67f6-0838-49d3-a4db-d22bbcd608d8 admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:08:27.888472 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb203373-b2cc-4387-8e4e-b3e064534141 None None] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:08:27.893357 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afc540a7-f38e-424d-90cf-db0472de8f3b service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f436f07939d4855a21b195bc7566244', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:08:27.894003 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afc540a7-f38e-424d-90cf-db0472de8f3b service neutron] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:08:27.894284 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afc540a7-f38e-424d-90cf-db0472de8f3b service neutron] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:08:27.894557 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afc540a7-f38e-424d-90cf-db0472de8f3b service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:08:27.898736 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1025/2082] 10.222.0.50 () {58 vars in 1122 bytes} [Tue Sep 13 16:08:27 2022] GET /identity/v3/roles => generated 1420 bytes in 40 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 16:08:27.903235 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e38f67f6-0838-49d3-a4db-d22bbcd608d8 admin admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:08:27.906213 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afc540a7-f38e-424d-90cf-db0472de8f3b service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:08:27.907631 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afc540a7-f38e-424d-90cf-db0472de8f3b service neutron] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:08:27.917134 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e38f67f6-0838-49d3-a4db-d22bbcd608d8 admin admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:08:27.919272 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1058/2083] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 16:08:27 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 104 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 16:08:27.924137 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db480a64-3c3f-406a-b790-67f98b9d6c83 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:08:27.924987 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db480a64-3c3f-406a-b790-67f98b9d6c83 admin admin] REQUEST_METHOD: `PUT` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:08:27.925270 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db480a64-3c3f-406a-b790-67f98b9d6c83 admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:08:27.925543 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db480a64-3c3f-406a-b790-67f98b9d6c83 admin admin] PATH_INFO: `/v3/projects/56de78ec0bde4337810e7256a0a336a9/users/3ea52f0cf89242a89849a8df377f5b17/roles/031d4e1a23e34f71858fc4a6dfbd4ccb` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:08:27.944094 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db480a64-3c3f-406a-b790-67f98b9d6c83 admin admin] RBAC: Authorizing `identity:create_grant(project_id=56de78ec0bde4337810e7256a0a336a9, user_id=3ea52f0cf89242a89849a8df377f5b17, role_id=031d4e1a23e34f71858fc4a6dfbd4ccb)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:08:27.949320 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db480a64-3c3f-406a-b790-67f98b9d6c83 admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:08:27.961858 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:08:27.968537 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1026/2084] 10.222.0.50 () {60 vars in 1483 bytes} [Tue Sep 13 16:08:27 2022] PUT /identity/v3/projects/56de78ec0bde4337810e7256a0a336a9/users/3ea52f0cf89242a89849a8df377f5b17/roles/031d4e1a23e34f71858fc4a6dfbd4ccb => generated 0 bytes in 66 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 13 16:08:27.973121 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afc540a7-f38e-424d-90cf-db0472de8f3b service neutron] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:08:27.999852 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afc540a7-f38e-424d-90cf-db0472de8f3b service neutron] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:08:28.096723 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c125b162-f7d7-4f97-a853-8cff0323a17a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:08:28.097591 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c125b162-f7d7-4f97-a853-8cff0323a17a admin admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:08:28.098172 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c125b162-f7d7-4f97-a853-8cff0323a17a admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:08:28.098800 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c125b162-f7d7-4f97-a853-8cff0323a17a admin admin] PATH_INFO: `/v3/roles` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:08:28.099789 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c125b162-f7d7-4f97-a853-8cff0323a17a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:08:28.109988 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c125b162-f7d7-4f97-a853-8cff0323a17a admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:08:28.114360 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1059/2085] 10.222.0.50 () {58 vars in 1122 bytes} [Tue Sep 13 16:08:27 2022] GET /identity/v3/roles => generated 1420 bytes in 142 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 16:08:28.119568 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db480a64-3c3f-406a-b790-67f98b9d6c83 admin admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:08:28.132847 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db480a64-3c3f-406a-b790-67f98b9d6c83 admin admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:08:28.138210 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-807fd076-8227-4502-9fb2-9549ff07129a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:08:28.139462 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-807fd076-8227-4502-9fb2-9549ff07129a admin admin] REQUEST_METHOD: `PUT` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:08:28.139584 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-807fd076-8227-4502-9fb2-9549ff07129a admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:08:28.139948 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-807fd076-8227-4502-9fb2-9549ff07129a admin admin] PATH_INFO: `/v3/projects/56de78ec0bde4337810e7256a0a336a9/users/3ea52f0cf89242a89849a8df377f5b17/roles/031d4e1a23e34f71858fc4a6dfbd4ccb` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:08:28.142125 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-807fd076-8227-4502-9fb2-9549ff07129a admin admin] RBAC: Authorizing `identity:create_grant(project_id=56de78ec0bde4337810e7256a0a336a9, user_id=3ea52f0cf89242a89849a8df377f5b17, role_id=031d4e1a23e34f71858fc4a6dfbd4ccb)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:08:28.145036 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-807fd076-8227-4502-9fb2-9549ff07129a admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:08:28.157098 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1027/2086] 10.222.0.50 () {60 vars in 1483 bytes} [Tue Sep 13 16:08:28 2022] PUT /identity/v3/projects/56de78ec0bde4337810e7256a0a336a9/users/3ea52f0cf89242a89849a8df377f5b17/roles/031d4e1a23e34f71858fc4a6dfbd4ccb => generated 0 bytes in 38 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 16:08:28.168829 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f65ee74-c63d-47f5-802a-21da1820a2a4 None None] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:08:28.170055 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f65ee74-c63d-47f5-802a-21da1820a2a4 None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:08:28.170055 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f65ee74-c63d-47f5-802a-21da1820a2a4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:08:28.200529 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.auth.core [None req-3f65ee74-c63d-47f5-802a-21da1820a2a4 None None] MFA Rules not processed for user `3ea52f0cf89242a89849a8df377f5b17`. Rule list: `[]` (Enabled: `True`). {{(pid=81437) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:08:28.201740 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-3f65ee74-c63d-47f5-802a-21da1820a2a4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:08:28.283000 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1060/2087] 10.222.0.50 () {58 vars in 958 bytes} [Tue Sep 13 16:08:28 2022] POST /identity/v3/auth/tokens => generated 3321 bytes in 115 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:08:28.295400 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-807fd076-8227-4502-9fb2-9549ff07129a admin admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:08:28.297071 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-807fd076-8227-4502-9fb2-9549ff07129a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:08:28.363884 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-807fd076-8227-4502-9fb2-9549ff07129a admin admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:08:28.368964 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d5c9222-898a-415e-bc39-8641c9260ce7 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7fd6b5afd67b48f9b0f0597ccd4cf4df', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:08:28.369654 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d5c9222-898a-415e-bc39-8641c9260ce7 service cinder] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:08:28.369999 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d5c9222-898a-415e-bc39-8641c9260ce7 service cinder] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:08:28.370338 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d5c9222-898a-415e-bc39-8641c9260ce7 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:08:28.380362 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d5c9222-898a-415e-bc39-8641c9260ce7 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:08:28.382133 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d5c9222-898a-415e-bc39-8641c9260ce7 service cinder] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:08:28.394117 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1028/2088] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 16:08:28 2022] GET /identity/v3/auth/tokens => generated 3321 bytes in 99 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 16:08:28.441645 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f65ee74-c63d-47f5-802a-21da1820a2a4 None None] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:08:28.443811 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-3f65ee74-c63d-47f5-802a-21da1820a2a4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:08:28.456718 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f65ee74-c63d-47f5-802a-21da1820a2a4 None None] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:08:28.460282 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d5c9222-898a-415e-bc39-8641c9260ce7 service cinder] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:08:28.461848 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-0d5c9222-898a-415e-bc39-8641c9260ce7 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:08:28.463314 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef0965c2-59fb-41eb-bfa8-1dfc228669c2 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7fd6b5afd67b48f9b0f0597ccd4cf4df', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:08:28.464032 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef0965c2-59fb-41eb-bfa8-1dfc228669c2 service cinder] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:08:28.464423 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef0965c2-59fb-41eb-bfa8-1dfc228669c2 service cinder] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:08:28.464796 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef0965c2-59fb-41eb-bfa8-1dfc228669c2 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:08:28.466614 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-ef0965c2-59fb-41eb-bfa8-1dfc228669c2 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:08:28.491743 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:08:28.547179 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef0965c2-59fb-41eb-bfa8-1dfc228669c2 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:08:28.548641 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef0965c2-59fb-41eb-bfa8-1dfc228669c2 service cinder] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:08:28.558037 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d5c9222-898a-415e-bc39-8641c9260ce7 service cinder] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:08:28.559804 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1061/2089] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 16:08:28 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 118 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 16:08:28.565622 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c316dfe8-c250-4777-b106-8319408da59d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:08:28.566261 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c316dfe8-c250-4777-b106-8319408da59d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:08:28.566536 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c316dfe8-c250-4777-b106-8319408da59d admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:08:28.566805 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c316dfe8-c250-4777-b106-8319408da59d admin admin] PATH_INFO: `/v3/projects/ab1ec172a97348899465eaeba821aad9` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:08:28.569257 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c316dfe8-c250-4777-b106-8319408da59d admin admin] RBAC: Authorizing `identity:delete_project(project_id=ab1ec172a97348899465eaeba821aad9)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:08:28.571811 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c316dfe8-c250-4777-b106-8319408da59d admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:08:28.632467 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-c316dfe8-c250-4777-b106-8319408da59d admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'ab1ec172a97348899465eaeba821aad9', 'request_id': 'req-c316dfe8-c250-4777-b106-8319408da59d'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:08:28.642548 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-c316dfe8-c250-4777-b106-8319408da59d admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'ab1ec172a97348899465eaeba821aad9', 'request_id': 'req-c316dfe8-c250-4777-b106-8319408da59d'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:08:28.649989 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-c316dfe8-c250-4777-b106-8319408da59d admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'ab1ec172a97348899465eaeba821aad9', 'request_id': 'req-c316dfe8-c250-4777-b106-8319408da59d'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:08:28.662737 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-c316dfe8-c250-4777-b106-8319408da59d admin admin] The token cache is being invalidate because project ab1ec172a97348899465eaeba821aad9 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81438) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 13 16:08:28.663393 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-c316dfe8-c250-4777-b106-8319408da59d admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:08:28.663737 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-c316dfe8-c250-4777-b106-8319408da59d admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:08:28.667255 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1029/2090] 10.222.0.50 () {58 vars in 1233 bytes} [Tue Sep 13 16:08:28 2022] DELETE /identity/v3/projects/ab1ec172a97348899465eaeba821aad9 => generated 0 bytes in 206 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 13 16:08:28.670566 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef0965c2-59fb-41eb-bfa8-1dfc228669c2 service cinder] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:08:28.683222 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef0965c2-59fb-41eb-bfa8-1dfc228669c2 service cinder] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:08:28.755814 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b3577ff-08ea-4f40-91d7-9de700db8ecb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:08:28.756479 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b3577ff-08ea-4f40-91d7-9de700db8ecb admin admin] REQUEST_METHOD: `DELETE` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:08:28.756762 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b3577ff-08ea-4f40-91d7-9de700db8ecb admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:08:28.757040 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b3577ff-08ea-4f40-91d7-9de700db8ecb admin admin] PATH_INFO: `/v3/users/53f033391a5d46d0a973b8ddb0681b5e` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:08:28.758224 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b3577ff-08ea-4f40-91d7-9de700db8ecb admin admin] RBAC: Authorizing `identity:delete_user(user_id=53f033391a5d46d0a973b8ddb0681b5e)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:08:28.760415 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b3577ff-08ea-4f40-91d7-9de700db8ecb admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:08:28.867458 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-2b3577ff-08ea-4f40-91d7-9de700db8ecb admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '53f033391a5d46d0a973b8ddb0681b5e'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:08:28.894623 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-2b3577ff-08ea-4f40-91d7-9de700db8ecb admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '53f033391a5d46d0a973b8ddb0681b5e'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:08:28.927882 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-2b3577ff-08ea-4f40-91d7-9de700db8ecb admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '53f033391a5d46d0a973b8ddb0681b5e'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:08:28.929211 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-2b3577ff-08ea-4f40-91d7-9de700db8ecb admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '53f033391a5d46d0a973b8ddb0681b5e'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:08:28.939988 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-2b3577ff-08ea-4f40-91d7-9de700db8ecb admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '53f033391a5d46d0a973b8ddb0681b5e'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:08:28.950278 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1062/2091] 10.222.0.50 () {58 vars in 1224 bytes} [Tue Sep 13 16:08:28 2022] DELETE /identity/v3/users/53f033391a5d46d0a973b8ddb0681b5e => generated 0 bytes in 278 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 16:08:29.348554 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c316dfe8-c250-4777-b106-8319408da59d admin admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:08:29.350132 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-c316dfe8-c250-4777-b106-8319408da59d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:08:29.365177 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c316dfe8-c250-4777-b106-8319408da59d admin admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:08:29.390686 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=91,keystone:INSERT=2,keystone:DELETE=7 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:08:29.434380 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-747d0dcb-cb68-46ec-bc48-806f9316823d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:08:29.435066 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-747d0dcb-cb68-46ec-bc48-806f9316823d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:08:29.435346 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-747d0dcb-cb68-46ec-bc48-806f9316823d admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:08:29.435617 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-747d0dcb-cb68-46ec-bc48-806f9316823d admin admin] PATH_INFO: `/v3/projects/f1ef7af8595e401abaf4564a5e6d2085` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:08:29.436738 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-747d0dcb-cb68-46ec-bc48-806f9316823d admin admin] RBAC: Authorizing `identity:delete_project(project_id=f1ef7af8595e401abaf4564a5e6d2085)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:08:29.438632 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-747d0dcb-cb68-46ec-bc48-806f9316823d admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:08:29.492357 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-747d0dcb-cb68-46ec-bc48-806f9316823d admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'f1ef7af8595e401abaf4564a5e6d2085', 'request_id': 'req-747d0dcb-cb68-46ec-bc48-806f9316823d'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:08:29.498888 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-747d0dcb-cb68-46ec-bc48-806f9316823d admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'f1ef7af8595e401abaf4564a5e6d2085', 'request_id': 'req-747d0dcb-cb68-46ec-bc48-806f9316823d'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:08:29.505592 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-747d0dcb-cb68-46ec-bc48-806f9316823d admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'f1ef7af8595e401abaf4564a5e6d2085', 'request_id': 'req-747d0dcb-cb68-46ec-bc48-806f9316823d'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:08:29.517881 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-747d0dcb-cb68-46ec-bc48-806f9316823d admin admin] The token cache is being invalidate because project f1ef7af8595e401abaf4564a5e6d2085 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81438) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 13 16:08:29.518171 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-747d0dcb-cb68-46ec-bc48-806f9316823d admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:08:29.518838 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-747d0dcb-cb68-46ec-bc48-806f9316823d admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:08:29.521012 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1030/2092] 10.222.0.50 () {58 vars in 1233 bytes} [Tue Sep 13 16:08:29 2022] DELETE /identity/v3/projects/f1ef7af8595e401abaf4564a5e6d2085 => generated 0 bytes in 174 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 16:08:29.525750 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ce7e879-b740-4e8f-8570-0272e065fc6a None None] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:08:29.526037 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ce7e879-b740-4e8f-8570-0272e065fc6a None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:08:29.526318 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ce7e879-b740-4e8f-8570-0272e065fc6a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:08:29.549205 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.auth.core [None req-8ce7e879-b740-4e8f-8570-0272e065fc6a None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81437) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:08:29.550448 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-8ce7e879-b740-4e8f-8570-0272e065fc6a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:08:29.584292 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:08:29.617081 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1063/2093] 10.222.0.50 () {58 vars in 958 bytes} [Tue Sep 13 16:08:29 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:08:29.624957 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1082c44f-86f2-4ac7-b6c8-8f3e13732b72 None None] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:08:29.625243 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1082c44f-86f2-4ac7-b6c8-8f3e13732b72 None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:08:29.625521 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1082c44f-86f2-4ac7-b6c8-8f3e13732b72 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:08:29.646115 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.auth.core [None req-1082c44f-86f2-4ac7-b6c8-8f3e13732b72 None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81438) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:08:29.647274 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-1082c44f-86f2-4ac7-b6c8-8f3e13732b72 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:08:29.655433 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1031/2094] 10.222.0.50 () {58 vars in 958 bytes} [Tue Sep 13 16:08:29 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:08:29.659596 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ce7e879-b740-4e8f-8570-0272e065fc6a None None] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:08:29.670229 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ce7e879-b740-4e8f-8570-0272e065fc6a None None] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:08:29.675274 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ab1ccfa-c557-48e4-870d-c6547e698ae2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:08:29.675876 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ab1ccfa-c557-48e4-870d-c6547e698ae2 admin admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:08:29.676233 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ab1ccfa-c557-48e4-870d-c6547e698ae2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:08:29.676979 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ab1ccfa-c557-48e4-870d-c6547e698ae2 admin admin] PATH_INFO: `/v3/domains` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:08:29.677879 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ab1ccfa-c557-48e4-870d-c6547e698ae2 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81437) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 16:08:29.678184 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ab1ccfa-c557-48e4-870d-c6547e698ae2 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:08:29.679926 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ab1ccfa-c557-48e4-870d-c6547e698ae2 admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:08:29.692408 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1064/2095] 10.222.0.50 () {58 vars in 1153 bytes} [Tue Sep 13 16:08:29 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 16:08:29.696933 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1082c44f-86f2-4ac7-b6c8-8f3e13732b72 None None] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:08:29.706686 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1082c44f-86f2-4ac7-b6c8-8f3e13732b72 None None] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:08:29.711581 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5839b06-95ec-4d47-9391-7ae066172a13 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:08:29.712191 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5839b06-95ec-4d47-9391-7ae066172a13 admin admin] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:08:29.712542 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5839b06-95ec-4d47-9391-7ae066172a13 admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:08:29.712911 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5839b06-95ec-4d47-9391-7ae066172a13 admin admin] PATH_INFO: `/v3/projects` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:08:29.713593 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5839b06-95ec-4d47-9391-7ae066172a13 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:08:29.715496 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5839b06-95ec-4d47-9391-7ae066172a13 admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:08:29.746382 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1032/2096] 10.222.0.50 () {60 vars in 1153 bytes} [Tue Sep 13 16:08:29 2022] POST /identity/v3/projects => generated 373 bytes in 50 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 16:08:29.750480 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ab1ccfa-c557-48e4-870d-c6547e698ae2 admin admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:08:29.761351 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ab1ccfa-c557-48e4-870d-c6547e698ae2 admin admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:08:29.827121 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29fd8ca0-9236-4d45-8ac7-c33ba11d0aa1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:08:29.827734 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29fd8ca0-9236-4d45-8ac7-c33ba11d0aa1 admin admin] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:08:29.828017 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29fd8ca0-9236-4d45-8ac7-c33ba11d0aa1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:08:29.828336 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29fd8ca0-9236-4d45-8ac7-c33ba11d0aa1 admin admin] PATH_INFO: `/v3/users` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:08:29.828943 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29fd8ca0-9236-4d45-8ac7-c33ba11d0aa1 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:08:29.830870 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29fd8ca0-9236-4d45-8ac7-c33ba11d0aa1 admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:08:29.853214 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1065/2097] 10.222.0.50 () {60 vars in 1144 bytes} [Tue Sep 13 16:08:29 2022] POST /identity/v3/users => generated 287 bytes in 103 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 16:08:29.857466 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5839b06-95ec-4d47-9391-7ae066172a13 admin admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:08:29.867656 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5839b06-95ec-4d47-9391-7ae066172a13 admin admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:08:29.872455 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f23e7467-a187-4f5c-8e07-65c61da7b234 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:08:29.873061 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f23e7467-a187-4f5c-8e07-65c61da7b234 admin admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:08:29.873340 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f23e7467-a187-4f5c-8e07-65c61da7b234 admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:08:29.873612 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f23e7467-a187-4f5c-8e07-65c61da7b234 admin admin] PATH_INFO: `/v3/roles` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:08:29.874246 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f23e7467-a187-4f5c-8e07-65c61da7b234 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:08:29.875881 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f23e7467-a187-4f5c-8e07-65c61da7b234 admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:08:29.886413 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1033/2098] 10.222.0.50 () {58 vars in 1122 bytes} [Tue Sep 13 16:08:29 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 16:08:29.890325 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29fd8ca0-9236-4d45-8ac7-c33ba11d0aa1 admin admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:08:29.900962 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29fd8ca0-9236-4d45-8ac7-c33ba11d0aa1 admin admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:08:29.905687 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae1cac09-9af5-4cae-8f2a-a36f8a46eefb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:08:29.906308 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae1cac09-9af5-4cae-8f2a-a36f8a46eefb admin admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:08:29.906591 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae1cac09-9af5-4cae-8f2a-a36f8a46eefb admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:08:29.906865 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae1cac09-9af5-4cae-8f2a-a36f8a46eefb admin admin] PATH_INFO: `/v3/roles` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:08:29.907539 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae1cac09-9af5-4cae-8f2a-a36f8a46eefb admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:08:29.909135 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae1cac09-9af5-4cae-8f2a-a36f8a46eefb admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:08:29.920369 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1066/2099] 10.222.0.50 () {58 vars in 1122 bytes} [Tue Sep 13 16:08:29 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 16:08:29.924308 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f23e7467-a187-4f5c-8e07-65c61da7b234 admin admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:08:29.934232 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f23e7467-a187-4f5c-8e07-65c61da7b234 admin admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:08:29.938843 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18de75cf-ba52-455c-83f3-d6b5dfca96cd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:08:29.939517 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18de75cf-ba52-455c-83f3-d6b5dfca96cd admin admin] REQUEST_METHOD: `PUT` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:08:29.939803 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18de75cf-ba52-455c-83f3-d6b5dfca96cd admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:08:29.940080 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18de75cf-ba52-455c-83f3-d6b5dfca96cd admin admin] PATH_INFO: `/v3/projects/04650bcf4da349c184b993cbaadb6871/users/9e14c4b1e41544f1862cd85c60091bc7/roles/15db4aaca3264ae29bde38903ba43471` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:08:29.954806 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18de75cf-ba52-455c-83f3-d6b5dfca96cd admin admin] RBAC: Authorizing `identity:create_grant(project_id=04650bcf4da349c184b993cbaadb6871, user_id=9e14c4b1e41544f1862cd85c60091bc7, role_id=15db4aaca3264ae29bde38903ba43471)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:08:29.958061 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18de75cf-ba52-455c-83f3-d6b5dfca96cd admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:08:29.972071 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1034/2100] 10.222.0.50 () {60 vars in 1483 bytes} [Tue Sep 13 16:08:29 2022] PUT /identity/v3/projects/04650bcf4da349c184b993cbaadb6871/users/9e14c4b1e41544f1862cd85c60091bc7/roles/15db4aaca3264ae29bde38903ba43471 => generated 0 bytes in 48 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 16:08:29.981741 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51a0f3b7-ba7e-4d03-87e9-750792c3f91e None None] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:08:29.981741 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51a0f3b7-ba7e-4d03-87e9-750792c3f91e None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:08:29.981741 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51a0f3b7-ba7e-4d03-87e9-750792c3f91e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:08:29.999266 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.auth.core [None req-51a0f3b7-ba7e-4d03-87e9-750792c3f91e None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81437) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:08:30.000465 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-51a0f3b7-ba7e-4d03-87e9-750792c3f91e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:08:30.062410 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1067/2101] 10.222.0.50 () {58 vars in 958 bytes} [Tue Sep 13 16:08:29 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:08:30.074174 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18de75cf-ba52-455c-83f3-d6b5dfca96cd admin admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:08:30.075959 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-18de75cf-ba52-455c-83f3-d6b5dfca96cd admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:08:30.131141 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:08:30.137584 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18de75cf-ba52-455c-83f3-d6b5dfca96cd admin admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:08:30.142797 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fce9cb97-3453-4944-b16f-4112a727e2e4 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f436f07939d4855a21b195bc7566244', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:08:30.143902 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fce9cb97-3453-4944-b16f-4112a727e2e4 service neutron] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:08:30.144331 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fce9cb97-3453-4944-b16f-4112a727e2e4 service neutron] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:08:30.145463 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fce9cb97-3453-4944-b16f-4112a727e2e4 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:08:30.159373 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fce9cb97-3453-4944-b16f-4112a727e2e4 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:08:30.161319 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fce9cb97-3453-4944-b16f-4112a727e2e4 service neutron] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:08:30.170365 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fe64ec5-b38b-4074-9b0a-b5ff4f36ee7d None None] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:08:30.170649 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fe64ec5-b38b-4074-9b0a-b5ff4f36ee7d None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:08:30.170943 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fe64ec5-b38b-4074-9b0a-b5ff4f36ee7d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:08:30.174847 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1035/2102] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 16:08:30 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 101 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 13 16:08:30.198158 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.auth.core [None req-2fe64ec5-b38b-4074-9b0a-b5ff4f36ee7d None None] MFA Rules not processed for user `f8364b1a190d4218bf353a89bdb81448`. Rule list: `[]` (Enabled: `True`). {{(pid=81437) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:08:30.200977 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-2fe64ec5-b38b-4074-9b0a-b5ff4f36ee7d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:08:30.256623 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:08:30.294067 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1068/2103] 10.222.0.50 () {58 vars in 1009 bytes} [Tue Sep 13 16:08:30 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 125 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 13 16:08:30.352453 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fce9cb97-3453-4944-b16f-4112a727e2e4 service neutron] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:08:30.354360 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-fce9cb97-3453-4944-b16f-4112a727e2e4 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:08:30.485844 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fce9cb97-3453-4944-b16f-4112a727e2e4 service neutron] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:08:30.493241 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59bf0f02-a6b9-4177-b5b0-53aaed2522fc service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b00ab0d6321d483188bc69deaddcdbbc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:08:30.493960 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59bf0f02-a6b9-4177-b5b0-53aaed2522fc service swift] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:08:30.494477 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59bf0f02-a6b9-4177-b5b0-53aaed2522fc service swift] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:08:30.494756 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59bf0f02-a6b9-4177-b5b0-53aaed2522fc service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:08:30.507296 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59bf0f02-a6b9-4177-b5b0-53aaed2522fc service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:08:30.509268 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59bf0f02-a6b9-4177-b5b0-53aaed2522fc service swift] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:08:30.523202 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1036/2104] 10.222.0.50 () {60 vars in 1384 bytes} [Tue Sep 13 16:08:30 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 171 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Sep 13 16:08:31.026590 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3037a428-914c-4f79-9f59-748e1fee3e11 None None] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:08:31.027138 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3037a428-914c-4f79-9f59-748e1fee3e11 None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:08:31.027248 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3037a428-914c-4f79-9f59-748e1fee3e11 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:08:31.090196 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.auth.core [None req-3037a428-914c-4f79-9f59-748e1fee3e11 None None] MFA Rules not processed for user `9e14c4b1e41544f1862cd85c60091bc7`. Rule list: `[]` (Enabled: `True`). {{(pid=81437) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:08:31.101352 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-3037a428-914c-4f79-9f59-748e1fee3e11 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:08:31.201412 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1069/2105] 10.222.0.50 () {58 vars in 958 bytes} [Tue Sep 13 16:08:31 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 176 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:08:31.205848 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59bf0f02-a6b9-4177-b5b0-53aaed2522fc service swift] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:08:31.217568 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59bf0f02-a6b9-4177-b5b0-53aaed2522fc service swift] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:08:31.234464 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82bf94d9-7235-43b5-b0d0-e800546eedbd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:08:31.235269 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82bf94d9-7235-43b5-b0d0-e800546eedbd admin admin] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:08:31.235618 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82bf94d9-7235-43b5-b0d0-e800546eedbd admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:08:31.235972 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82bf94d9-7235-43b5-b0d0-e800546eedbd admin admin] PATH_INFO: `/v3/projects` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:08:31.236695 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82bf94d9-7235-43b5-b0d0-e800546eedbd admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:08:31.239497 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82bf94d9-7235-43b5-b0d0-e800546eedbd admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:08:31.277536 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1037/2106] 10.222.0.50 () {60 vars in 1153 bytes} [Tue Sep 13 16:08:31 2022] POST /identity/v3/projects => generated 373 bytes in 72 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 16:08:31.281606 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3037a428-914c-4f79-9f59-748e1fee3e11 None None] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:08:31.294665 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3037a428-914c-4f79-9f59-748e1fee3e11 None None] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:08:31.327045 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fdec3d4-03ed-44c1-8dc1-49e71ca0abe1 None None] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:08:31.327418 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fdec3d4-03ed-44c1-8dc1-49e71ca0abe1 None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:08:31.327766 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fdec3d4-03ed-44c1-8dc1-49e71ca0abe1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:08:31.360602 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bf223d3-93de-4001-a2a5-30837cd8bbc5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:08:31.361207 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bf223d3-93de-4001-a2a5-30837cd8bbc5 admin admin] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:08:31.361486 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bf223d3-93de-4001-a2a5-30837cd8bbc5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:08:31.361760 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bf223d3-93de-4001-a2a5-30837cd8bbc5 admin admin] PATH_INFO: `/v3/users` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:08:31.362401 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4bf223d3-93de-4001-a2a5-30837cd8bbc5 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:08:31.364447 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.auth.core [None req-3fdec3d4-03ed-44c1-8dc1-49e71ca0abe1 None None] MFA Rules not processed for user `f8364b1a190d4218bf353a89bdb81448`. Rule list: `[]` (Enabled: `True`). {{(pid=81438) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:08:31.364605 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4bf223d3-93de-4001-a2a5-30837cd8bbc5 admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:08:31.365954 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-3fdec3d4-03ed-44c1-8dc1-49e71ca0abe1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:08:31.393909 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1070/2107] 10.222.0.50 () {60 vars in 1144 bytes} [Tue Sep 13 16:08:31 2022] POST /identity/v3/users => generated 293 bytes in 111 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 16:08:31.398619 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bf223d3-93de-4001-a2a5-30837cd8bbc5 admin admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:08:31.410221 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bf223d3-93de-4001-a2a5-30837cd8bbc5 admin admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:08:31.416186 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0510127-5a0c-4d95-9691-905dd3c8e0f9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:08:31.416800 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0510127-5a0c-4d95-9691-905dd3c8e0f9 admin admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:08:31.417352 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0510127-5a0c-4d95-9691-905dd3c8e0f9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:08:31.417352 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0510127-5a0c-4d95-9691-905dd3c8e0f9 admin admin] PATH_INFO: `/v3/roles` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:08:31.418009 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0510127-5a0c-4d95-9691-905dd3c8e0f9 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:08:31.419809 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0510127-5a0c-4d95-9691-905dd3c8e0f9 admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:08:31.432248 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1071/2108] 10.222.0.50 () {58 vars in 1122 bytes} [Tue Sep 13 16:08:31 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 13 16:08:31.436110 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0510127-5a0c-4d95-9691-905dd3c8e0f9 admin admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:08:31.437921 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1038/2109] 10.222.0.50 () {58 vars in 967 bytes} [Tue Sep 13 16:08:31 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 112 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 13 16:08:31.446032 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fdec3d4-03ed-44c1-8dc1-49e71ca0abe1 None None] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:08:31.449974 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0510127-5a0c-4d95-9691-905dd3c8e0f9 admin admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:08:31.454916 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb37ec00-b1fe-43a2-bb7d-e2503cb63fbc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:08:31.455583 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb37ec00-b1fe-43a2-bb7d-e2503cb63fbc admin admin] REQUEST_METHOD: `PUT` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:08:31.456160 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb37ec00-b1fe-43a2-bb7d-e2503cb63fbc admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:08:31.456160 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb37ec00-b1fe-43a2-bb7d-e2503cb63fbc admin admin] PATH_INFO: `/v3/projects/d3262fc47ea44842a292beb228bd256d/users/05db30cc72f74957b378a05672507630/roles/031d4e1a23e34f71858fc4a6dfbd4ccb` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:08:31.476514 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb37ec00-b1fe-43a2-bb7d-e2503cb63fbc admin admin] RBAC: Authorizing `identity:create_grant(project_id=d3262fc47ea44842a292beb228bd256d, user_id=05db30cc72f74957b378a05672507630, role_id=031d4e1a23e34f71858fc4a6dfbd4ccb)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:08:31.479530 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb37ec00-b1fe-43a2-bb7d-e2503cb63fbc admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:08:31.495893 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1072/2110] 10.222.0.50 () {60 vars in 1483 bytes} [Tue Sep 13 16:08:31 2022] PUT /identity/v3/projects/d3262fc47ea44842a292beb228bd256d/users/05db30cc72f74957b378a05672507630/roles/031d4e1a23e34f71858fc4a6dfbd4ccb => generated 0 bytes in 60 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 16:08:31.499764 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb37ec00-b1fe-43a2-bb7d-e2503cb63fbc admin admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:08:31.526394 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:08:31.530698 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fdec3d4-03ed-44c1-8dc1-49e71ca0abe1 None None] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:08:31.534074 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:08:31.576519 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb37ec00-b1fe-43a2-bb7d-e2503cb63fbc admin admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:08:31.583082 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62e3574a-8896-45cb-973f-b88f4b27a086 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:08:31.584008 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62e3574a-8896-45cb-973f-b88f4b27a086 admin admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:08:31.584452 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62e3574a-8896-45cb-973f-b88f4b27a086 admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:08:31.584729 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62e3574a-8896-45cb-973f-b88f4b27a086 admin admin] PATH_INFO: `/v3/roles` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:08:31.585380 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62e3574a-8896-45cb-973f-b88f4b27a086 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:08:31.587164 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62e3574a-8896-45cb-973f-b88f4b27a086 admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:08:31.600000 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1073/2111] 10.222.0.50 () {58 vars in 1122 bytes} [Tue Sep 13 16:08:31 2022] GET /identity/v3/roles => generated 1420 bytes in 100 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 13 16:08:31.604722 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62e3574a-8896-45cb-973f-b88f4b27a086 admin admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:08:31.616654 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62e3574a-8896-45cb-973f-b88f4b27a086 admin admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:08:31.621963 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d17587b6-d306-4dac-a49f-e406ecf21ed2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:08:31.622627 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d17587b6-d306-4dac-a49f-e406ecf21ed2 admin admin] REQUEST_METHOD: `PUT` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:08:31.622921 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d17587b6-d306-4dac-a49f-e406ecf21ed2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:08:31.623194 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d17587b6-d306-4dac-a49f-e406ecf21ed2 admin admin] PATH_INFO: `/v3/projects/d3262fc47ea44842a292beb228bd256d/users/05db30cc72f74957b378a05672507630/roles/031d4e1a23e34f71858fc4a6dfbd4ccb` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:08:31.626062 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d17587b6-d306-4dac-a49f-e406ecf21ed2 admin admin] RBAC: Authorizing `identity:create_grant(project_id=d3262fc47ea44842a292beb228bd256d, user_id=05db30cc72f74957b378a05672507630, role_id=031d4e1a23e34f71858fc4a6dfbd4ccb)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:08:31.629654 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d17587b6-d306-4dac-a49f-e406ecf21ed2 admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:08:31.644440 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c0d8fdd-c93a-4ebe-96f8-b80bb8f7d922 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b00ab0d6321d483188bc69deaddcdbbc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:08:31.645990 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c0d8fdd-c93a-4ebe-96f8-b80bb8f7d922 service swift] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:08:31.646103 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1074/2112] 10.222.0.50 () {60 vars in 1483 bytes} [Tue Sep 13 16:08:31 2022] PUT /identity/v3/projects/d3262fc47ea44842a292beb228bd256d/users/05db30cc72f74957b378a05672507630/roles/031d4e1a23e34f71858fc4a6dfbd4ccb => generated 0 bytes in 42 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 16:08:31.646447 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c0d8fdd-c93a-4ebe-96f8-b80bb8f7d922 service swift] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:08:31.646735 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c0d8fdd-c93a-4ebe-96f8-b80bb8f7d922 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:08:31.719286 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c0d8fdd-c93a-4ebe-96f8-b80bb8f7d922 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:08:31.720952 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c0d8fdd-c93a-4ebe-96f8-b80bb8f7d922 service swift] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:08:31.731162 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1039/2113] 10.222.0.50 () {60 vars in 1384 bytes} [Tue Sep 13 16:08:31 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 285 msecs (HTTP/1.1 200) 6 headers in 379 bytes (2 switches on core 0) Sep 13 16:08:31.865974 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d17587b6-d306-4dac-a49f-e406ecf21ed2 admin admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:08:31.868948 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-d17587b6-d306-4dac-a49f-e406ecf21ed2 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:08:31.948732 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d17587b6-d306-4dac-a49f-e406ecf21ed2 admin admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:08:31.953752 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03225154-564c-4043-a503-0ac252ff2635 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:08:31.954415 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03225154-564c-4043-a503-0ac252ff2635 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:08:31.954629 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03225154-564c-4043-a503-0ac252ff2635 admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:08:31.955012 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03225154-564c-4043-a503-0ac252ff2635 admin admin] PATH_INFO: `/v3/users/ea071cd9cef94e2e94cc4b63f4488680` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:08:31.956964 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03225154-564c-4043-a503-0ac252ff2635 admin admin] RBAC: Authorizing `identity:delete_user(user_id=ea071cd9cef94e2e94cc4b63f4488680)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:08:31.959819 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03225154-564c-4043-a503-0ac252ff2635 admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:08:32.020569 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d68432a-8699-4dfe-bb31-6c4d8e4c8841 None None] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:08:32.020859 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d68432a-8699-4dfe-bb31-6c4d8e4c8841 None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:08:32.021189 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d68432a-8699-4dfe-bb31-6c4d8e4c8841 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:08:32.029602 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-03225154-564c-4043-a503-0ac252ff2635 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'ea071cd9cef94e2e94cc4b63f4488680'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:08:32.042835 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-03225154-564c-4043-a503-0ac252ff2635 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'ea071cd9cef94e2e94cc4b63f4488680'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:08:32.053628 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.auth.core [None req-1d68432a-8699-4dfe-bb31-6c4d8e4c8841 None None] MFA Rules not processed for user `f8364b1a190d4218bf353a89bdb81448`. Rule list: `[]` (Enabled: `True`). {{(pid=81438) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:08:32.055066 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-1d68432a-8699-4dfe-bb31-6c4d8e4c8841 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:08:32.089437 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-03225154-564c-4043-a503-0ac252ff2635 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'ea071cd9cef94e2e94cc4b63f4488680'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:08:32.099049 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-03225154-564c-4043-a503-0ac252ff2635 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'ea071cd9cef94e2e94cc4b63f4488680'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:08:32.115386 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-03225154-564c-4043-a503-0ac252ff2635 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'ea071cd9cef94e2e94cc4b63f4488680'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:08:32.132275 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1075/2114] 10.222.0.50 () {58 vars in 1224 bytes} [Tue Sep 13 16:08:31 2022] DELETE /identity/v3/users/ea071cd9cef94e2e94cc4b63f4488680 => generated 0 bytes in 262 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 16:08:32.135186 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-402aafd1-9277-4aba-bc77-5e61a605ef00 None None] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:08:32.135933 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-402aafd1-9277-4aba-bc77-5e61a605ef00 None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:08:32.136415 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-402aafd1-9277-4aba-bc77-5e61a605ef00 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:08:32.159184 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1040/2115] 10.222.0.50 () {58 vars in 1009 bytes} [Tue Sep 13 16:08:32 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 140 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:08:32.171135 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.auth.core [None req-402aafd1-9277-4aba-bc77-5e61a605ef00 None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81437) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:08:32.172254 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-402aafd1-9277-4aba-bc77-5e61a605ef00 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:08:32.176883 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d68432a-8699-4dfe-bb31-6c4d8e4c8841 None None] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:08:32.178742 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-1d68432a-8699-4dfe-bb31-6c4d8e4c8841 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:08:32.207121 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=88,keystone:INSERT=2,keystone:DELETE=10 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:08:32.255094 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1076/2116] 10.222.0.50 () {58 vars in 958 bytes} [Tue Sep 13 16:08:32 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 122 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:08:32.266220 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-402aafd1-9277-4aba-bc77-5e61a605ef00 None None] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:08:32.268006 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-402aafd1-9277-4aba-bc77-5e61a605ef00 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:08:32.268266 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:08:32.299286 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d68432a-8699-4dfe-bb31-6c4d8e4c8841 None None] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:08:32.364444 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-402aafd1-9277-4aba-bc77-5e61a605ef00 None None] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:08:32.369415 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2831a67-38ee-4feb-81e0-fc72c2d1e532 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f436f07939d4855a21b195bc7566244', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:08:32.370160 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2831a67-38ee-4feb-81e0-fc72c2d1e532 service neutron] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:08:32.370496 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2831a67-38ee-4feb-81e0-fc72c2d1e532 service neutron] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:08:32.371562 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2831a67-38ee-4feb-81e0-fc72c2d1e532 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:08:32.382313 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2831a67-38ee-4feb-81e0-fc72c2d1e532 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:08:32.383966 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2831a67-38ee-4feb-81e0-fc72c2d1e532 service neutron] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:08:32.393020 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c292a50d-1412-474d-badf-dfa618385617 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b00ab0d6321d483188bc69deaddcdbbc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:08:32.393643 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c292a50d-1412-474d-badf-dfa618385617 service swift] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:08:32.393931 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c292a50d-1412-474d-badf-dfa618385617 service swift] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:08:32.394291 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c292a50d-1412-474d-badf-dfa618385617 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:08:32.397175 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1077/2117] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 16:08:32 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 131 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 13 16:08:32.477772 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c292a50d-1412-474d-badf-dfa618385617 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:08:32.480918 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c292a50d-1412-474d-badf-dfa618385617 service swift] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:08:32.499154 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1041/2118] 10.222.0.50 () {60 vars in 1384 bytes} [Tue Sep 13 16:08:32 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 323 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Sep 13 16:08:33.014400 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20e5449a-fc1e-47f1-8869-332898c65d27 None None] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:08:33.014934 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20e5449a-fc1e-47f1-8869-332898c65d27 None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:08:33.015072 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20e5449a-fc1e-47f1-8869-332898c65d27 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:08:33.044124 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.auth.core [None req-20e5449a-fc1e-47f1-8869-332898c65d27 None None] MFA Rules not processed for user `f8364b1a190d4218bf353a89bdb81448`. Rule list: `[]` (Enabled: `True`). {{(pid=81437) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:08:33.045991 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-20e5449a-fc1e-47f1-8869-332898c65d27 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:08:33.056645 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1078/2119] 10.222.0.50 () {58 vars in 967 bytes} [Tue Sep 13 16:08:33 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 44 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:08:33.078226 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c292a50d-1412-474d-badf-dfa618385617 service swift] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:08:33.096747 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c292a50d-1412-474d-badf-dfa618385617 service swift] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:08:33.105528 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb56f52e-ad2a-4a17-8dc8-cdcd27e31b64 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b00ab0d6321d483188bc69deaddcdbbc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:08:33.106291 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb56f52e-ad2a-4a17-8dc8-cdcd27e31b64 service swift] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:08:33.106677 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb56f52e-ad2a-4a17-8dc8-cdcd27e31b64 service swift] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:08:33.107232 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb56f52e-ad2a-4a17-8dc8-cdcd27e31b64 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:08:33.119176 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb56f52e-ad2a-4a17-8dc8-cdcd27e31b64 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:08:33.121859 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb56f52e-ad2a-4a17-8dc8-cdcd27e31b64 service swift] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:08:33.134846 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1042/2120] 10.222.0.50 () {60 vars in 1384 bytes} [Tue Sep 13 16:08:33 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 58 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Sep 13 16:08:33.205768 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20e5449a-fc1e-47f1-8869-332898c65d27 None None] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:08:33.208862 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-20e5449a-fc1e-47f1-8869-332898c65d27 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:08:33.241377 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20e5449a-fc1e-47f1-8869-332898c65d27 None None] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:08:33.252324 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f453ee33-06c4-4ea4-a056-9f25e1768be5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:08:33.253291 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f453ee33-06c4-4ea4-a056-9f25e1768be5 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:08:33.253610 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f453ee33-06c4-4ea4-a056-9f25e1768be5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:08:33.254015 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f453ee33-06c4-4ea4-a056-9f25e1768be5 admin admin] PATH_INFO: `/v3/projects/d1b4f10d46fa43a7bd0619c3839ca8da` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:08:33.255950 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f453ee33-06c4-4ea4-a056-9f25e1768be5 admin admin] RBAC: Authorizing `identity:delete_project(project_id=d1b4f10d46fa43a7bd0619c3839ca8da)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:08:33.259888 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f453ee33-06c4-4ea4-a056-9f25e1768be5 admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:08:33.276481 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7422a6d8-b36e-4f47-b784-021d5f0c6577 None None] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:08:33.277057 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7422a6d8-b36e-4f47-b784-021d5f0c6577 None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:08:33.277267 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7422a6d8-b36e-4f47-b784-021d5f0c6577 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:08:33.339033 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.auth.core [None req-7422a6d8-b36e-4f47-b784-021d5f0c6577 None None] MFA Rules not processed for user `05db30cc72f74957b378a05672507630`. Rule list: `[]` (Enabled: `True`). {{(pid=81438) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:08:33.340630 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-7422a6d8-b36e-4f47-b784-021d5f0c6577 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:08:33.344525 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-f453ee33-06c4-4ea4-a056-9f25e1768be5 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'd1b4f10d46fa43a7bd0619c3839ca8da', 'request_id': 'req-f453ee33-06c4-4ea4-a056-9f25e1768be5'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:08:33.367082 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-f453ee33-06c4-4ea4-a056-9f25e1768be5 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'd1b4f10d46fa43a7bd0619c3839ca8da', 'request_id': 'req-f453ee33-06c4-4ea4-a056-9f25e1768be5'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:08:33.393221 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-f453ee33-06c4-4ea4-a056-9f25e1768be5 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'd1b4f10d46fa43a7bd0619c3839ca8da', 'request_id': 'req-f453ee33-06c4-4ea4-a056-9f25e1768be5'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:08:33.428829 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-f453ee33-06c4-4ea4-a056-9f25e1768be5 admin admin] The token cache is being invalidate because project d1b4f10d46fa43a7bd0619c3839ca8da was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81437) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 13 16:08:33.429300 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-f453ee33-06c4-4ea4-a056-9f25e1768be5 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:08:33.429997 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-f453ee33-06c4-4ea4-a056-9f25e1768be5 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:08:33.449374 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1079/2121] 10.222.0.50 () {58 vars in 1233 bytes} [Tue Sep 13 16:08:33 2022] DELETE /identity/v3/projects/d1b4f10d46fa43a7bd0619c3839ca8da => generated 0 bytes in 240 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 16:08:33.450717 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f453ee33-06c4-4ea4-a056-9f25e1768be5 admin admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:08:33.453728 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-f453ee33-06c4-4ea4-a056-9f25e1768be5 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:08:33.498694 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1043/2122] 10.222.0.50 () {58 vars in 958 bytes} [Tue Sep 13 16:08:33 2022] POST /identity/v3/auth/tokens => generated 3317 bytes in 224 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 13 16:08:33.541686 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7422a6d8-b36e-4f47-b784-021d5f0c6577 None None] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:08:33.547687 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-7422a6d8-b36e-4f47-b784-021d5f0c6577 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:08:33.564183 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:08:33.578015 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:08:33.600893 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f453ee33-06c4-4ea4-a056-9f25e1768be5 admin admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:08:33.606683 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28d420cd-ce00-43d1-bcca-c446892f230d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:08:33.607599 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28d420cd-ce00-43d1-bcca-c446892f230d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:08:33.607955 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28d420cd-ce00-43d1-bcca-c446892f230d admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:08:33.608363 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28d420cd-ce00-43d1-bcca-c446892f230d admin admin] PATH_INFO: `/v3/users/6978e9d2bd45491db3aaa7fd58e6f250` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:08:33.610019 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28d420cd-ce00-43d1-bcca-c446892f230d admin admin] RBAC: Authorizing `identity:delete_user(user_id=6978e9d2bd45491db3aaa7fd58e6f250)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:08:33.612976 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28d420cd-ce00-43d1-bcca-c446892f230d admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:08:33.681763 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7422a6d8-b36e-4f47-b784-021d5f0c6577 None None] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:08:33.695253 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fcf8140-7302-4a78-82ff-eb971ca16dc3 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7fd6b5afd67b48f9b0f0597ccd4cf4df', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:08:33.695951 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fcf8140-7302-4a78-82ff-eb971ca16dc3 service cinder] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:08:33.696431 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fcf8140-7302-4a78-82ff-eb971ca16dc3 service cinder] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:08:33.697667 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fcf8140-7302-4a78-82ff-eb971ca16dc3 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:08:33.701419 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-4fcf8140-7302-4a78-82ff-eb971ca16dc3 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:08:33.703416 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-28d420cd-ce00-43d1-bcca-c446892f230d admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '6978e9d2bd45491db3aaa7fd58e6f250'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:08:33.719790 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-28d420cd-ce00-43d1-bcca-c446892f230d admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '6978e9d2bd45491db3aaa7fd58e6f250'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:08:33.736571 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-28d420cd-ce00-43d1-bcca-c446892f230d admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '6978e9d2bd45491db3aaa7fd58e6f250'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:08:33.737565 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-28d420cd-ce00-43d1-bcca-c446892f230d admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '6978e9d2bd45491db3aaa7fd58e6f250'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:08:33.747834 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-28d420cd-ce00-43d1-bcca-c446892f230d admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '6978e9d2bd45491db3aaa7fd58e6f250'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:08:33.759143 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1080/2123] 10.222.0.50 () {58 vars in 1224 bytes} [Tue Sep 13 16:08:33 2022] DELETE /identity/v3/users/6978e9d2bd45491db3aaa7fd58e6f250 => generated 0 bytes in 309 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 16:08:33.864481 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4fcf8140-7302-4a78-82ff-eb971ca16dc3 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:08:33.867643 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4fcf8140-7302-4a78-82ff-eb971ca16dc3 service cinder] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:08:33.887018 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1044/2124] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 16:08:33 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 346 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 13 16:08:34.301818 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28d420cd-ce00-43d1-bcca-c446892f230d admin admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:08:34.303277 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-28d420cd-ce00-43d1-bcca-c446892f230d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:08:34.483218 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28d420cd-ce00-43d1-bcca-c446892f230d admin admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:08:34.489724 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e2bb564-2e06-4a82-947d-8b2f8e97018f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:08:34.490543 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e2bb564-2e06-4a82-947d-8b2f8e97018f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:08:34.491011 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e2bb564-2e06-4a82-947d-8b2f8e97018f admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:08:34.491108 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e2bb564-2e06-4a82-947d-8b2f8e97018f admin admin] PATH_INFO: `/v3/projects/2ca361e398514a9f919ae53bd82a60ac` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:08:34.500798 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e2bb564-2e06-4a82-947d-8b2f8e97018f admin admin] RBAC: Authorizing `identity:delete_project(project_id=2ca361e398514a9f919ae53bd82a60ac)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:08:34.504010 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e2bb564-2e06-4a82-947d-8b2f8e97018f admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:08:34.567692 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-9e2bb564-2e06-4a82-947d-8b2f8e97018f admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '2ca361e398514a9f919ae53bd82a60ac', 'request_id': 'req-9e2bb564-2e06-4a82-947d-8b2f8e97018f'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:08:34.577335 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-9e2bb564-2e06-4a82-947d-8b2f8e97018f admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '2ca361e398514a9f919ae53bd82a60ac', 'request_id': 'req-9e2bb564-2e06-4a82-947d-8b2f8e97018f'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:08:34.585967 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-9e2bb564-2e06-4a82-947d-8b2f8e97018f admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '2ca361e398514a9f919ae53bd82a60ac', 'request_id': 'req-9e2bb564-2e06-4a82-947d-8b2f8e97018f'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:08:34.593747 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=81,keystone:DELETE=17,keystone:INSERT=2 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:08:34.599285 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-9e2bb564-2e06-4a82-947d-8b2f8e97018f admin admin] The token cache is being invalidate because project 2ca361e398514a9f919ae53bd82a60ac was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81437) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 13 16:08:34.600048 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-9e2bb564-2e06-4a82-947d-8b2f8e97018f admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:08:34.601305 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-9e2bb564-2e06-4a82-947d-8b2f8e97018f admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:08:34.606871 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1081/2125] 10.222.0.50 () {58 vars in 1233 bytes} [Tue Sep 13 16:08:34 2022] DELETE /identity/v3/projects/2ca361e398514a9f919ae53bd82a60ac => generated 0 bytes in 303 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 16:08:34.607813 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fcf8140-7302-4a78-82ff-eb971ca16dc3 service cinder] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:08:34.623806 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fcf8140-7302-4a78-82ff-eb971ca16dc3 service cinder] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:08:34.743388 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6afbf06b-55d6-4a4d-a443-c0dd219ffb76 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:08:34.744149 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6afbf06b-55d6-4a4d-a443-c0dd219ffb76 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:08:34.745094 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6afbf06b-55d6-4a4d-a443-c0dd219ffb76 admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:08:34.745463 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6afbf06b-55d6-4a4d-a443-c0dd219ffb76 admin admin] PATH_INFO: `/v3/users/3ea52f0cf89242a89849a8df377f5b17` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:08:34.746968 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6afbf06b-55d6-4a4d-a443-c0dd219ffb76 admin admin] RBAC: Authorizing `identity:delete_user(user_id=3ea52f0cf89242a89849a8df377f5b17)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:08:34.752280 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6afbf06b-55d6-4a4d-a443-c0dd219ffb76 admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:08:34.846487 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-6afbf06b-55d6-4a4d-a443-c0dd219ffb76 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '3ea52f0cf89242a89849a8df377f5b17'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:08:34.855615 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-6afbf06b-55d6-4a4d-a443-c0dd219ffb76 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '3ea52f0cf89242a89849a8df377f5b17'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:08:34.865059 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=90,keystone:DELETE=9,keystone:INSERT=1 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:08:34.881499 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-6afbf06b-55d6-4a4d-a443-c0dd219ffb76 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '3ea52f0cf89242a89849a8df377f5b17'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:08:34.881499 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-6afbf06b-55d6-4a4d-a443-c0dd219ffb76 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '3ea52f0cf89242a89849a8df377f5b17'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:08:34.898723 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-6afbf06b-55d6-4a4d-a443-c0dd219ffb76 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '3ea52f0cf89242a89849a8df377f5b17'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:08:34.905430 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1045/2126] 10.222.0.50 () {58 vars in 1224 bytes} [Tue Sep 13 16:08:34 2022] DELETE /identity/v3/users/3ea52f0cf89242a89849a8df377f5b17 => generated 0 bytes in 298 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 16:08:35.464405 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e2bb564-2e06-4a82-947d-8b2f8e97018f admin admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:08:35.465892 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-9e2bb564-2e06-4a82-947d-8b2f8e97018f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:08:35.609756 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e2bb564-2e06-4a82-947d-8b2f8e97018f admin admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:08:35.677506 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df16c0e9-f433-422c-b560-6e9666bdfad4 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7fd6b5afd67b48f9b0f0597ccd4cf4df', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:08:35.678836 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df16c0e9-f433-422c-b560-6e9666bdfad4 service cinder] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:08:35.680014 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df16c0e9-f433-422c-b560-6e9666bdfad4 service cinder] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:08:35.680448 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df16c0e9-f433-422c-b560-6e9666bdfad4 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:08:35.686376 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-df16c0e9-f433-422c-b560-6e9666bdfad4 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:08:35.699747 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6afbf06b-55d6-4a4d-a443-c0dd219ffb76 admin admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:08:35.702582 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-6afbf06b-55d6-4a4d-a443-c0dd219ffb76 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:08:35.809892 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6afbf06b-55d6-4a4d-a443-c0dd219ffb76 admin admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:08:35.816231 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df16c0e9-f433-422c-b560-6e9666bdfad4 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:08:35.817116 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afe7e082-bf12-4d23-9e5e-084ae8d29ce3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:08:35.817797 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afe7e082-bf12-4d23-9e5e-084ae8d29ce3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:08:35.818080 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afe7e082-bf12-4d23-9e5e-084ae8d29ce3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:08:35.818354 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afe7e082-bf12-4d23-9e5e-084ae8d29ce3 admin admin] PATH_INFO: `/v3/projects/56de78ec0bde4337810e7256a0a336a9` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:08:35.819037 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df16c0e9-f433-422c-b560-6e9666bdfad4 service cinder] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:08:35.819579 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afe7e082-bf12-4d23-9e5e-084ae8d29ce3 admin admin] RBAC: Authorizing `identity:delete_project(project_id=56de78ec0bde4337810e7256a0a336a9)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:08:35.821843 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afe7e082-bf12-4d23-9e5e-084ae8d29ce3 admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:08:35.847161 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1082/2127] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 16:08:35 2022] GET /identity/v3/auth/tokens => generated 3317 bytes in 383 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 16:08:35.906317 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-afe7e082-bf12-4d23-9e5e-084ae8d29ce3 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '56de78ec0bde4337810e7256a0a336a9', 'request_id': 'req-afe7e082-bf12-4d23-9e5e-084ae8d29ce3'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:08:35.915105 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-afe7e082-bf12-4d23-9e5e-084ae8d29ce3 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '56de78ec0bde4337810e7256a0a336a9', 'request_id': 'req-afe7e082-bf12-4d23-9e5e-084ae8d29ce3'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:08:35.923875 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-afe7e082-bf12-4d23-9e5e-084ae8d29ce3 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '56de78ec0bde4337810e7256a0a336a9', 'request_id': 'req-afe7e082-bf12-4d23-9e5e-084ae8d29ce3'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:08:35.936880 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-afe7e082-bf12-4d23-9e5e-084ae8d29ce3 admin admin] The token cache is being invalidate because project 56de78ec0bde4337810e7256a0a336a9 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81438) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 13 16:08:35.937333 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-afe7e082-bf12-4d23-9e5e-084ae8d29ce3 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:08:35.938071 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-afe7e082-bf12-4d23-9e5e-084ae8d29ce3 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:08:35.939937 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1046/2128] 10.222.0.50 () {58 vars in 1233 bytes} [Tue Sep 13 16:08:35 2022] DELETE /identity/v3/projects/56de78ec0bde4337810e7256a0a336a9 => generated 0 bytes in 242 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 16:08:35.946494 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa8935ac-41ce-4e7e-9a8a-7488f3d3aa61 None None] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:08:35.946964 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa8935ac-41ce-4e7e-9a8a-7488f3d3aa61 None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:08:35.947344 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa8935ac-41ce-4e7e-9a8a-7488f3d3aa61 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:08:35.969379 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.auth.core [None req-fa8935ac-41ce-4e7e-9a8a-7488f3d3aa61 None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81437) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:08:35.970603 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-fa8935ac-41ce-4e7e-9a8a-7488f3d3aa61 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:08:36.046326 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1083/2129] 10.222.0.50 () {58 vars in 958 bytes} [Tue Sep 13 16:08:35 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 13 16:08:36.057017 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1817891-1d37-4a5e-9e02-57ed2d593654 None None] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:08:36.057391 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1817891-1d37-4a5e-9e02-57ed2d593654 None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:08:36.057628 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1817891-1d37-4a5e-9e02-57ed2d593654 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:08:36.084324 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.auth.core [None req-a1817891-1d37-4a5e-9e02-57ed2d593654 None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81438) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:08:36.085717 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-a1817891-1d37-4a5e-9e02-57ed2d593654 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:08:36.095130 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1047/2130] 10.222.0.50 () {58 vars in 958 bytes} [Tue Sep 13 16:08:36 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 39 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 13 16:08:36.100243 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa8935ac-41ce-4e7e-9a8a-7488f3d3aa61 None None] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:08:36.115119 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa8935ac-41ce-4e7e-9a8a-7488f3d3aa61 None None] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:08:36.120290 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7bd0839-b18f-4523-bc93-cd51cc540894 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:08:36.120905 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7bd0839-b18f-4523-bc93-cd51cc540894 admin admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:08:36.121175 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7bd0839-b18f-4523-bc93-cd51cc540894 admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:08:36.121443 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7bd0839-b18f-4523-bc93-cd51cc540894 admin admin] PATH_INFO: `/v3/domains` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:08:36.122084 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7bd0839-b18f-4523-bc93-cd51cc540894 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81437) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 16:08:36.122381 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7bd0839-b18f-4523-bc93-cd51cc540894 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:08:36.124000 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7bd0839-b18f-4523-bc93-cd51cc540894 admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:08:36.125817 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:08:36.128321 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a43bf95e-d2e2-4132-97df-49739b1fd34c None None] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:08:36.129272 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a43bf95e-d2e2-4132-97df-49739b1fd34c None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:08:36.129763 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a43bf95e-d2e2-4132-97df-49739b1fd34c None None] PATH_INFO: `/` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:08:36.131915 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1048/2131] 10.222.0.50 () {54 vars in 921 bytes} [Tue Sep 13 16:08:36 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 16:08:36.139347 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1084/2132] 10.222.0.50 () {58 vars in 1153 bytes} [Tue Sep 13 16:08:36 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 40 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 13 16:08:36.143669 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a43bf95e-d2e2-4132-97df-49739b1fd34c None None] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:08:36.157646 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a43bf95e-d2e2-4132-97df-49739b1fd34c None None] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:08:36.163465 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45d62a8a-87ea-4455-8d32-215efbba692f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:08:36.164293 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45d62a8a-87ea-4455-8d32-215efbba692f admin admin] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:08:36.164589 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5681123-8513-456d-b2ee-56ac4f4953df None None] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:08:36.164799 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45d62a8a-87ea-4455-8d32-215efbba692f admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:08:36.165030 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5681123-8513-456d-b2ee-56ac4f4953df None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:08:36.165414 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5681123-8513-456d-b2ee-56ac4f4953df None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:08:36.165546 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45d62a8a-87ea-4455-8d32-215efbba692f admin admin] PATH_INFO: `/v3/projects` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:08:36.166065 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45d62a8a-87ea-4455-8d32-215efbba692f admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:08:36.168060 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45d62a8a-87ea-4455-8d32-215efbba692f admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:08:36.191934 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.auth.core [None req-f5681123-8513-456d-b2ee-56ac4f4953df None None] MFA Rules not processed for user `31480b61418340bd88583f034f67412e`. Rule list: `[]` (Enabled: `True`). {{(pid=81437) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:08:36.193521 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-f5681123-8513-456d-b2ee-56ac4f4953df None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:08:36.220510 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1049/2133] 10.222.0.50 () {60 vars in 1153 bytes} [Tue Sep 13 16:08:36 2022] POST /identity/v3/projects => generated 381 bytes in 77 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 16:08:36.221362 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45d62a8a-87ea-4455-8d32-215efbba692f admin admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:08:36.228570 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-45d62a8a-87ea-4455-8d32-215efbba692f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:08:36.280510 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=90,keystone:DELETE=8,keystone:INSERT=2 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:08:36.282706 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1085/2134] 10.222.0.50 () {58 vars in 1018 bytes} [Tue Sep 13 16:08:36 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 119 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 13 16:08:36.283592 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5681123-8513-456d-b2ee-56ac4f4953df None None] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:08:36.285438 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-f5681123-8513-456d-b2ee-56ac4f4953df None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:08:36.299237 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5681123-8513-456d-b2ee-56ac4f4953df None None] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:08:36.311238 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45d62a8a-87ea-4455-8d32-215efbba692f admin admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:08:36.316397 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae4b6fae-9c48-4958-8e6f-2ef0a632ded6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:08:36.317049 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae4b6fae-9c48-4958-8e6f-2ef0a632ded6 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:08:36.317328 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae4b6fae-9c48-4958-8e6f-2ef0a632ded6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:08:36.318042 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae4b6fae-9c48-4958-8e6f-2ef0a632ded6 admin admin] PATH_INFO: `/v3/users/96751dfa79154d3885eca3e37648c2a9` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:08:36.319245 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae4b6fae-9c48-4958-8e6f-2ef0a632ded6 admin admin] RBAC: Authorizing `identity:delete_user(user_id=96751dfa79154d3885eca3e37648c2a9)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:08:36.321136 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae4b6fae-9c48-4958-8e6f-2ef0a632ded6 admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:08:36.358661 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e62f7438-b18d-43b4-8b47-b26fc0aeda1f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:08:36.359343 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e62f7438-b18d-43b4-8b47-b26fc0aeda1f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:08:36.359875 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e62f7438-b18d-43b4-8b47-b26fc0aeda1f admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:08:36.360209 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e62f7438-b18d-43b4-8b47-b26fc0aeda1f admin admin] PATH_INFO: `/v3/users/179e2ae27b924db0a49b64502548525a` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:08:36.361409 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e62f7438-b18d-43b4-8b47-b26fc0aeda1f admin admin] RBAC: Authorizing `identity:delete_user(user_id=179e2ae27b924db0a49b64502548525a)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:08:36.364012 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e62f7438-b18d-43b4-8b47-b26fc0aeda1f admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:08:36.370396 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-ae4b6fae-9c48-4958-8e6f-2ef0a632ded6 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '96751dfa79154d3885eca3e37648c2a9'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:08:36.378784 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-ae4b6fae-9c48-4958-8e6f-2ef0a632ded6 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '96751dfa79154d3885eca3e37648c2a9'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:08:36.391634 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-ae4b6fae-9c48-4958-8e6f-2ef0a632ded6 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '96751dfa79154d3885eca3e37648c2a9'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:08:36.392327 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-ae4b6fae-9c48-4958-8e6f-2ef0a632ded6 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '96751dfa79154d3885eca3e37648c2a9'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:08:36.400219 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-ae4b6fae-9c48-4958-8e6f-2ef0a632ded6 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '96751dfa79154d3885eca3e37648c2a9'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:08:36.406398 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1050/2135] 10.222.0.50 () {58 vars in 1224 bytes} [Tue Sep 13 16:08:36 2022] DELETE /identity/v3/users/96751dfa79154d3885eca3e37648c2a9 => generated 0 bytes in 185 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 16:08:36.407482 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae4b6fae-9c48-4958-8e6f-2ef0a632ded6 admin admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:08:36.408819 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-ae4b6fae-9c48-4958-8e6f-2ef0a632ded6 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:08:36.416783 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-e62f7438-b18d-43b4-8b47-b26fc0aeda1f admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '179e2ae27b924db0a49b64502548525a'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:08:36.424335 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-e62f7438-b18d-43b4-8b47-b26fc0aeda1f admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '179e2ae27b924db0a49b64502548525a'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:08:36.437776 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-e62f7438-b18d-43b4-8b47-b26fc0aeda1f admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '179e2ae27b924db0a49b64502548525a'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:08:36.438533 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-e62f7438-b18d-43b4-8b47-b26fc0aeda1f admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '179e2ae27b924db0a49b64502548525a'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:08:36.445644 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-e62f7438-b18d-43b4-8b47-b26fc0aeda1f admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '179e2ae27b924db0a49b64502548525a'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:08:36.451648 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1086/2136] 10.222.0.50 () {58 vars in 1224 bytes} [Tue Sep 13 16:08:36 2022] DELETE /identity/v3/users/179e2ae27b924db0a49b64502548525a => generated 0 bytes in 168 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 16:08:36.455551 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e62f7438-b18d-43b4-8b47-b26fc0aeda1f admin admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:08:36.456906 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-e62f7438-b18d-43b4-8b47-b26fc0aeda1f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:08:36.467505 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:08:36.480339 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae4b6fae-9c48-4958-8e6f-2ef0a632ded6 admin admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:08:36.537028 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e62f7438-b18d-43b4-8b47-b26fc0aeda1f admin admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:08:36.544743 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c5f9fb5-e9b2-45c0-b92d-f0fcf5754fd2 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '31480b61418340bd88583f034f67412e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:08:36.545351 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c5f9fb5-e9b2-45c0-b92d-f0fcf5754fd2 service placement] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:08:36.545629 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c5f9fb5-e9b2-45c0-b92d-f0fcf5754fd2 service placement] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:08:36.545905 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c5f9fb5-e9b2-45c0-b92d-f0fcf5754fd2 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:08:36.559791 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-9c5f9fb5-e9b2-45c0-b92d-f0fcf5754fd2 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:08:36.561818 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37f019f4-db7c-4aca-b0b1-66a3a209f9da admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:08:36.562377 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37f019f4-db7c-4aca-b0b1-66a3a209f9da admin admin] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:08:36.562668 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37f019f4-db7c-4aca-b0b1-66a3a209f9da admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:08:36.563054 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37f019f4-db7c-4aca-b0b1-66a3a209f9da admin admin] PATH_INFO: `/v3/users` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:08:36.563691 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37f019f4-db7c-4aca-b0b1-66a3a209f9da admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:08:36.565606 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37f019f4-db7c-4aca-b0b1-66a3a209f9da admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:08:36.572442 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c5f9fb5-e9b2-45c0-b92d-f0fcf5754fd2 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:08:36.574183 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c5f9fb5-e9b2-45c0-b92d-f0fcf5754fd2 service placement] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:08:36.583677 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=86,keystone:DELETE=10,keystone:INSERT=4 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:08:36.585894 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1087/2137] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 16:08:36 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 134 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 16:08:36.587141 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43db1c8d-9416-4206-a4bc-eaaf7cce0b42 None None] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:08:36.587719 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43db1c8d-9416-4206-a4bc-eaaf7cce0b42 None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:08:36.587830 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43db1c8d-9416-4206-a4bc-eaaf7cce0b42 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:08:36.595198 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1051/2138] 10.222.0.50 () {60 vars in 1144 bytes} [Tue Sep 13 16:08:36 2022] POST /identity/v3/users => generated 291 bytes in 188 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 16:08:36.596040 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37f019f4-db7c-4aca-b0b1-66a3a209f9da admin admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:08:36.599293 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-37f019f4-db7c-4aca-b0b1-66a3a209f9da admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:08:36.611809 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37f019f4-db7c-4aca-b0b1-66a3a209f9da admin admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:08:36.616797 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.auth.core [None req-43db1c8d-9416-4206-a4bc-eaaf7cce0b42 None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81437) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:08:36.617250 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8a8b5b2-5754-4a08-ad3e-dac831f81f18 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:08:36.617860 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8a8b5b2-5754-4a08-ad3e-dac831f81f18 admin admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:08:36.618304 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8a8b5b2-5754-4a08-ad3e-dac831f81f18 admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:08:36.618588 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8a8b5b2-5754-4a08-ad3e-dac831f81f18 admin admin] PATH_INFO: `/v3/roles` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:08:36.619475 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-43db1c8d-9416-4206-a4bc-eaaf7cce0b42 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:08:36.619589 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8a8b5b2-5754-4a08-ad3e-dac831f81f18 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:08:36.621085 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8a8b5b2-5754-4a08-ad3e-dac831f81f18 admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:08:36.629165 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1088/2139] 10.222.0.50 () {58 vars in 958 bytes} [Tue Sep 13 16:08:36 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 42 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:08:36.633642 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1052/2140] 10.222.0.50 () {58 vars in 1122 bytes} [Tue Sep 13 16:08:36 2022] GET /identity/v3/roles => generated 1420 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 16:08:36.641685 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8a8b5b2-5754-4a08-ad3e-dac831f81f18 admin admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:08:36.641786 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43db1c8d-9416-4206-a4bc-eaaf7cce0b42 None None] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:08:36.643747 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-b8a8b5b2-5754-4a08-ad3e-dac831f81f18 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:08:36.659575 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43db1c8d-9416-4206-a4bc-eaaf7cce0b42 None None] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:08:36.681189 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b02d68a4-a788-4d3e-9a8d-40387d24973f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:08:36.682385 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b02d68a4-a788-4d3e-9a8d-40387d24973f admin admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:08:36.682670 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b02d68a4-a788-4d3e-9a8d-40387d24973f admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:08:36.682995 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b02d68a4-a788-4d3e-9a8d-40387d24973f admin admin] PATH_INFO: `/v3/roles` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:08:36.683650 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b02d68a4-a788-4d3e-9a8d-40387d24973f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:08:36.686752 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b02d68a4-a788-4d3e-9a8d-40387d24973f admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:08:36.720131 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1089/2141] 10.222.0.50 () {58 vars in 1122 bytes} [Tue Sep 13 16:08:36 2022] GET /identity/v3/roles => generated 1420 bytes in 79 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 16:08:36.721730 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b02d68a4-a788-4d3e-9a8d-40387d24973f admin admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:08:36.727161 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-b02d68a4-a788-4d3e-9a8d-40387d24973f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:08:36.768511 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8a8b5b2-5754-4a08-ad3e-dac831f81f18 admin admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:08:36.771083 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b02d68a4-a788-4d3e-9a8d-40387d24973f admin admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:08:36.776642 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e909f782-465e-4771-9f66-7a1fa61a016e service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f436f07939d4855a21b195bc7566244', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:08:36.776933 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06529f7f-b18d-4131-b0da-c35b71f5a938 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:08:36.777605 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06529f7f-b18d-4131-b0da-c35b71f5a938 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:08:36.777918 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06529f7f-b18d-4131-b0da-c35b71f5a938 admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:08:36.778299 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06529f7f-b18d-4131-b0da-c35b71f5a938 admin admin] PATH_INFO: `/v3/projects/a345ae605d56441e86b2dd45565655c5` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:08:36.778488 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e909f782-465e-4771-9f66-7a1fa61a016e service neutron] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:08:36.778990 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e909f782-465e-4771-9f66-7a1fa61a016e service neutron] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:08:36.779506 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e909f782-465e-4771-9f66-7a1fa61a016e service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:08:36.779634 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06529f7f-b18d-4131-b0da-c35b71f5a938 admin admin] RBAC: Authorizing `identity:delete_project(project_id=a345ae605d56441e86b2dd45565655c5)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:08:36.782150 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06529f7f-b18d-4131-b0da-c35b71f5a938 admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:08:36.797117 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e909f782-465e-4771-9f66-7a1fa61a016e service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:08:36.799490 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e909f782-465e-4771-9f66-7a1fa61a016e service neutron] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:08:36.810154 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1053/2142] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 16:08:36 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 169 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 16:08:36.810944 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e909f782-465e-4771-9f66-7a1fa61a016e service neutron] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:08:36.820928 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e909f782-465e-4771-9f66-7a1fa61a016e service neutron] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:08:36.825682 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-857f95d2-a83c-4f2e-8e0a-f587728c9d4a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:08:36.826394 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-857f95d2-a83c-4f2e-8e0a-f587728c9d4a admin admin] REQUEST_METHOD: `PUT` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:08:36.826681 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-857f95d2-a83c-4f2e-8e0a-f587728c9d4a admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:08:36.826984 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-857f95d2-a83c-4f2e-8e0a-f587728c9d4a admin admin] PATH_INFO: `/v3/projects/cdf8f2889bcf49b88b67c7ab6d872a9e/users/14f60a07cff34e8896d2ecb6c14a964e/roles/15db4aaca3264ae29bde38903ba43471` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:08:36.851309 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-857f95d2-a83c-4f2e-8e0a-f587728c9d4a admin admin] RBAC: Authorizing `identity:create_grant(project_id=cdf8f2889bcf49b88b67c7ab6d872a9e, user_id=14f60a07cff34e8896d2ecb6c14a964e, role_id=15db4aaca3264ae29bde38903ba43471)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:08:36.855377 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-857f95d2-a83c-4f2e-8e0a-f587728c9d4a admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:08:36.855796 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-06529f7f-b18d-4131-b0da-c35b71f5a938 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'a345ae605d56441e86b2dd45565655c5', 'request_id': 'req-06529f7f-b18d-4131-b0da-c35b71f5a938'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:08:36.881482 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1054/2143] 10.222.0.50 () {60 vars in 1483 bytes} [Tue Sep 13 16:08:36 2022] PUT /identity/v3/projects/cdf8f2889bcf49b88b67c7ab6d872a9e/users/14f60a07cff34e8896d2ecb6c14a964e/roles/15db4aaca3264ae29bde38903ba43471 => generated 0 bytes in 71 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 16:08:36.891339 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-06529f7f-b18d-4131-b0da-c35b71f5a938 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'a345ae605d56441e86b2dd45565655c5', 'request_id': 'req-06529f7f-b18d-4131-b0da-c35b71f5a938'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:08:36.895006 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdaf66cf-7a1e-41a5-bb15-db98af60b41e None None] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:08:36.895297 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdaf66cf-7a1e-41a5-bb15-db98af60b41e None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:08:36.895574 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdaf66cf-7a1e-41a5-bb15-db98af60b41e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:08:36.902829 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-06529f7f-b18d-4131-b0da-c35b71f5a938 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'a345ae605d56441e86b2dd45565655c5', 'request_id': 'req-06529f7f-b18d-4131-b0da-c35b71f5a938'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:08:36.923944 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-06529f7f-b18d-4131-b0da-c35b71f5a938 admin admin] The token cache is being invalidate because project a345ae605d56441e86b2dd45565655c5 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81437) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 13 16:08:36.924980 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-06529f7f-b18d-4131-b0da-c35b71f5a938 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:08:36.926379 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-06529f7f-b18d-4131-b0da-c35b71f5a938 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:08:36.929123 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1090/2144] 10.222.0.50 () {58 vars in 1233 bytes} [Tue Sep 13 16:08:36 2022] DELETE /identity/v3/projects/a345ae605d56441e86b2dd45565655c5 => generated 0 bytes in 208 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 16:08:36.940521 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.auth.core [None req-cdaf66cf-7a1e-41a5-bb15-db98af60b41e None None] MFA Rules not processed for user `14f60a07cff34e8896d2ecb6c14a964e`. Rule list: `[]` (Enabled: `True`). {{(pid=81438) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:08:36.942290 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-cdaf66cf-7a1e-41a5-bb15-db98af60b41e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:08:36.942719 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23a9fca7-709e-4630-bddd-fcc69fa3ce93 None None] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:08:36.943209 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23a9fca7-709e-4630-bddd-fcc69fa3ce93 None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:08:36.943665 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23a9fca7-709e-4630-bddd-fcc69fa3ce93 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:08:36.974270 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.auth.core [None req-23a9fca7-709e-4630-bddd-fcc69fa3ce93 None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81437) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:08:36.975779 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-23a9fca7-709e-4630-bddd-fcc69fa3ce93 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:08:37.019076 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1055/2145] 10.222.0.50 () {58 vars in 958 bytes} [Tue Sep 13 16:08:36 2022] POST /identity/v3/auth/tokens => generated 3258 bytes in 126 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 13 16:08:37.023082 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdaf66cf-7a1e-41a5-bb15-db98af60b41e None None] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:08:37.024664 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-cdaf66cf-7a1e-41a5-bb15-db98af60b41e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:08:37.054209 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1091/2146] 10.222.0.50 () {58 vars in 958 bytes} [Tue Sep 13 16:08:36 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 13 16:08:37.056171 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12182d2a-d058-4e8d-8350-43f910faf91e None None] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:08:37.056458 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12182d2a-d058-4e8d-8350-43f910faf91e None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:08:37.056926 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12182d2a-d058-4e8d-8350-43f910faf91e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:08:37.066713 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:08:37.080686 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.auth.core [None req-12182d2a-d058-4e8d-8350-43f910faf91e None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81437) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:08:37.083565 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-12182d2a-d058-4e8d-8350-43f910faf91e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:08:37.094024 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1092/2147] 10.222.0.50 () {58 vars in 958 bytes} [Tue Sep 13 16:08:37 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 39 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:08:37.098245 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12182d2a-d058-4e8d-8350-43f910faf91e None None] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:08:37.099406 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:08:37.106388 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdaf66cf-7a1e-41a5-bb15-db98af60b41e None None] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:08:37.107693 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12182d2a-d058-4e8d-8350-43f910faf91e None None] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:08:37.113067 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e549e005-2153-42cf-97ac-56660fa6d3d0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:08:37.113711 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e549e005-2153-42cf-97ac-56660fa6d3d0 admin admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:08:37.113978 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e549e005-2153-42cf-97ac-56660fa6d3d0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:08:37.114267 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e549e005-2153-42cf-97ac-56660fa6d3d0 admin admin] PATH_INFO: `/v3/domains` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:08:37.114849 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e549e005-2153-42cf-97ac-56660fa6d3d0 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81437) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 16:08:37.115169 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-459acbdf-9a5d-43af-bb5f-cb15b6012af5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:08:37.115290 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e549e005-2153-42cf-97ac-56660fa6d3d0 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:08:37.116085 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-459acbdf-9a5d-43af-bb5f-cb15b6012af5 admin admin] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:08:37.116414 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-459acbdf-9a5d-43af-bb5f-cb15b6012af5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:08:37.116795 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-459acbdf-9a5d-43af-bb5f-cb15b6012af5 admin admin] PATH_INFO: `/v3/projects` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:08:37.116909 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e549e005-2153-42cf-97ac-56660fa6d3d0 admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:08:37.118024 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-459acbdf-9a5d-43af-bb5f-cb15b6012af5 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:08:37.121171 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-459acbdf-9a5d-43af-bb5f-cb15b6012af5 admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:08:37.130641 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1093/2148] 10.222.0.50 () {58 vars in 1153 bytes} [Tue Sep 13 16:08:37 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 13 16:08:37.135394 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e549e005-2153-42cf-97ac-56660fa6d3d0 admin admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:08:37.148084 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e549e005-2153-42cf-97ac-56660fa6d3d0 admin admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:08:37.153291 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f545cad4-5269-41ad-a143-61b42e4e5d95 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:08:37.160487 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f545cad4-5269-41ad-a143-61b42e4e5d95 admin admin] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:08:37.160487 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f545cad4-5269-41ad-a143-61b42e4e5d95 admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:08:37.160487 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f545cad4-5269-41ad-a143-61b42e4e5d95 admin admin] PATH_INFO: `/v3/projects` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:08:37.160487 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f545cad4-5269-41ad-a143-61b42e4e5d95 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:08:37.160487 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f545cad4-5269-41ad-a143-61b42e4e5d95 admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:08:37.170044 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1056/2149] 10.222.0.50 () {60 vars in 1153 bytes} [Tue Sep 13 16:08:37 2022] POST /identity/v3/projects => generated 379 bytes in 147 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 13 16:08:37.170999 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-459acbdf-9a5d-43af-bb5f-cb15b6012af5 admin admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:08:37.172789 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-459acbdf-9a5d-43af-bb5f-cb15b6012af5 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:08:37.199354 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1094/2150] 10.222.0.50 () {60 vars in 1153 bytes} [Tue Sep 13 16:08:37 2022] POST /identity/v3/projects => generated 383 bytes in 64 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 13 16:08:37.200812 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f545cad4-5269-41ad-a143-61b42e4e5d95 admin admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:08:37.254328 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-459acbdf-9a5d-43af-bb5f-cb15b6012af5 admin admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:08:37.282703 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f545cad4-5269-41ad-a143-61b42e4e5d95 admin admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:08:37.289009 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-664c1f0f-3638-4817-8b94-b7fade65015e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:08:37.289630 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-664c1f0f-3638-4817-8b94-b7fade65015e admin admin] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:08:37.289991 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-664c1f0f-3638-4817-8b94-b7fade65015e admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:08:37.290265 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-664c1f0f-3638-4817-8b94-b7fade65015e admin admin] PATH_INFO: `/v3/users` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:08:37.290931 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-664c1f0f-3638-4817-8b94-b7fade65015e admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:08:37.292911 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-664c1f0f-3638-4817-8b94-b7fade65015e admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:08:37.316759 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1095/2151] 10.222.0.50 () {60 vars in 1144 bytes} [Tue Sep 13 16:08:37 2022] POST /identity/v3/users => generated 296 bytes in 117 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 16:08:37.317565 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-664c1f0f-3638-4817-8b94-b7fade65015e admin admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:08:37.322438 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38103762-583b-40c6-8bf0-2e5fb0abddf9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:08:37.323224 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38103762-583b-40c6-8bf0-2e5fb0abddf9 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:08:37.323743 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38103762-583b-40c6-8bf0-2e5fb0abddf9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:08:37.324024 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38103762-583b-40c6-8bf0-2e5fb0abddf9 admin admin] PATH_INFO: `/v3/projects/a19f8d067f67490cac7c827edcaaa8dc` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:08:37.325248 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38103762-583b-40c6-8bf0-2e5fb0abddf9 admin admin] RBAC: Authorizing `identity:delete_project(project_id=a19f8d067f67490cac7c827edcaaa8dc)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:08:37.327530 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38103762-583b-40c6-8bf0-2e5fb0abddf9 admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:08:37.328986 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-664c1f0f-3638-4817-8b94-b7fade65015e admin admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:08:37.334601 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ac5aa4b-2e31-4c3f-8ea9-abaff1a43ccc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:08:37.335237 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ac5aa4b-2e31-4c3f-8ea9-abaff1a43ccc admin admin] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:08:37.335516 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ac5aa4b-2e31-4c3f-8ea9-abaff1a43ccc admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:08:37.335790 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ac5aa4b-2e31-4c3f-8ea9-abaff1a43ccc admin admin] PATH_INFO: `/v3/users` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:08:37.336413 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ac5aa4b-2e31-4c3f-8ea9-abaff1a43ccc admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:08:37.338325 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ac5aa4b-2e31-4c3f-8ea9-abaff1a43ccc admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:08:37.363253 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1096/2152] 10.222.0.50 () {60 vars in 1144 bytes} [Tue Sep 13 16:08:37 2022] POST /identity/v3/users => generated 292 bytes in 46 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 16:08:37.364100 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ac5aa4b-2e31-4c3f-8ea9-abaff1a43ccc admin admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:08:37.379118 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ac5aa4b-2e31-4c3f-8ea9-abaff1a43ccc admin admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:08:37.387715 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-38103762-583b-40c6-8bf0-2e5fb0abddf9 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'a19f8d067f67490cac7c827edcaaa8dc', 'request_id': 'req-38103762-583b-40c6-8bf0-2e5fb0abddf9'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:08:37.397593 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-38103762-583b-40c6-8bf0-2e5fb0abddf9 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'a19f8d067f67490cac7c827edcaaa8dc', 'request_id': 'req-38103762-583b-40c6-8bf0-2e5fb0abddf9'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:08:37.405356 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-38103762-583b-40c6-8bf0-2e5fb0abddf9 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'a19f8d067f67490cac7c827edcaaa8dc', 'request_id': 'req-38103762-583b-40c6-8bf0-2e5fb0abddf9'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:08:37.420689 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-38103762-583b-40c6-8bf0-2e5fb0abddf9 admin admin] The token cache is being invalidate because project a19f8d067f67490cac7c827edcaaa8dc was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81438) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 13 16:08:37.421671 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-38103762-583b-40c6-8bf0-2e5fb0abddf9 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:08:37.422503 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-38103762-583b-40c6-8bf0-2e5fb0abddf9 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:08:37.424730 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1057/2153] 10.222.0.50 () {58 vars in 1233 bytes} [Tue Sep 13 16:08:37 2022] DELETE /identity/v3/projects/a19f8d067f67490cac7c827edcaaa8dc => generated 0 bytes in 254 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 16:08:37.426161 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38103762-583b-40c6-8bf0-2e5fb0abddf9 admin admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:08:37.428635 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-38103762-583b-40c6-8bf0-2e5fb0abddf9 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:08:37.484092 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a909889f-f2b3-48be-9701-df38904dc9cf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:08:37.484716 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a909889f-f2b3-48be-9701-df38904dc9cf admin admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:08:37.485218 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a909889f-f2b3-48be-9701-df38904dc9cf admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:08:37.485558 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a909889f-f2b3-48be-9701-df38904dc9cf admin admin] PATH_INFO: `/v3/roles` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:08:37.486361 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a909889f-f2b3-48be-9701-df38904dc9cf admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:08:37.489006 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a909889f-f2b3-48be-9701-df38904dc9cf admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:08:37.502068 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1097/2154] 10.222.0.50 () {58 vars in 1122 bytes} [Tue Sep 13 16:08:37 2022] GET /identity/v3/roles => generated 1420 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 16:08:37.504735 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=91,keystone:INSERT=2,keystone:DELETE=7 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:08:37.509154 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a909889f-f2b3-48be-9701-df38904dc9cf admin admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:08:37.513114 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-a909889f-f2b3-48be-9701-df38904dc9cf admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:08:37.531378 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a909889f-f2b3-48be-9701-df38904dc9cf admin admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:08:37.537815 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0ca3e41-1bb4-4106-9d83-59a92c7cac8f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:08:37.539323 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0ca3e41-1bb4-4106-9d83-59a92c7cac8f admin admin] REQUEST_METHOD: `PUT` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:08:37.539806 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0ca3e41-1bb4-4106-9d83-59a92c7cac8f admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:08:37.540183 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0ca3e41-1bb4-4106-9d83-59a92c7cac8f admin admin] PATH_INFO: `/v3/projects/2cf92f46cf6b41dabd418a33954b35ba/users/b72d1e75efc642acaa87c74a45e7ada8/roles/031d4e1a23e34f71858fc4a6dfbd4ccb` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:08:37.561767 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0ca3e41-1bb4-4106-9d83-59a92c7cac8f admin admin] RBAC: Authorizing `identity:create_grant(project_id=2cf92f46cf6b41dabd418a33954b35ba, user_id=b72d1e75efc642acaa87c74a45e7ada8, role_id=031d4e1a23e34f71858fc4a6dfbd4ccb)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:08:37.565336 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0ca3e41-1bb4-4106-9d83-59a92c7cac8f admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:08:37.575261 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38103762-583b-40c6-8bf0-2e5fb0abddf9 admin admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:08:37.581526 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba92e565-7f13-42c7-b719-7b9c9c6e9cd6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:08:37.582121 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba92e565-7f13-42c7-b719-7b9c9c6e9cd6 admin admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:08:37.582631 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba92e565-7f13-42c7-b719-7b9c9c6e9cd6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:08:37.582742 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba92e565-7f13-42c7-b719-7b9c9c6e9cd6 admin admin] PATH_INFO: `/v3/roles` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:08:37.583370 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba92e565-7f13-42c7-b719-7b9c9c6e9cd6 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:08:37.586038 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba92e565-7f13-42c7-b719-7b9c9c6e9cd6 admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:08:37.590099 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1098/2155] 10.222.0.50 () {60 vars in 1483 bytes} [Tue Sep 13 16:08:37 2022] PUT /identity/v3/projects/2cf92f46cf6b41dabd418a33954b35ba/users/b72d1e75efc642acaa87c74a45e7ada8/roles/031d4e1a23e34f71858fc4a6dfbd4ccb => generated 0 bytes in 83 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 13 16:08:37.593791 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0ca3e41-1bb4-4106-9d83-59a92c7cac8f admin admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:08:37.605416 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1058/2156] 10.222.0.50 () {58 vars in 1122 bytes} [Tue Sep 13 16:08:37 2022] GET /identity/v3/roles => generated 1420 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 16:08:37.607615 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=92,keystone:INSERT=8 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:08:37.608000 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba92e565-7f13-42c7-b719-7b9c9c6e9cd6 admin admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:08:37.620694 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba92e565-7f13-42c7-b719-7b9c9c6e9cd6 admin admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:08:37.670726 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0ca3e41-1bb4-4106-9d83-59a92c7cac8f admin admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:08:37.676793 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60af48f8-310f-4754-9b72-37ff6ca90788 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:08:37.677394 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60af48f8-310f-4754-9b72-37ff6ca90788 admin admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:08:37.677694 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60af48f8-310f-4754-9b72-37ff6ca90788 admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:08:37.677938 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60af48f8-310f-4754-9b72-37ff6ca90788 admin admin] PATH_INFO: `/v3/roles` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:08:37.678583 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60af48f8-310f-4754-9b72-37ff6ca90788 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:08:37.680281 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60af48f8-310f-4754-9b72-37ff6ca90788 admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:08:37.681267 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64125eb5-194e-4459-9453-9019632b64e2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:08:37.681909 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64125eb5-194e-4459-9453-9019632b64e2 admin admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:08:37.682186 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64125eb5-194e-4459-9453-9019632b64e2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:08:37.682462 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64125eb5-194e-4459-9453-9019632b64e2 admin admin] PATH_INFO: `/v3/roles` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:08:37.683113 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64125eb5-194e-4459-9453-9019632b64e2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:08:37.684681 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64125eb5-194e-4459-9453-9019632b64e2 admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:08:37.690935 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1099/2157] 10.222.0.50 () {58 vars in 1122 bytes} [Tue Sep 13 16:08:37 2022] GET /identity/v3/roles => generated 1420 bytes in 97 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 13 16:08:37.694791 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60af48f8-310f-4754-9b72-37ff6ca90788 admin admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:08:37.698050 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1059/2158] 10.222.0.50 () {58 vars in 1122 bytes} [Tue Sep 13 16:08:37 2022] GET /identity/v3/roles => generated 1420 bytes in 88 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 16:08:37.699158 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64125eb5-194e-4459-9453-9019632b64e2 admin admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:08:37.704531 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60af48f8-310f-4754-9b72-37ff6ca90788 admin admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:08:37.709661 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64125eb5-194e-4459-9453-9019632b64e2 admin admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:08:37.710194 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a41bfbf-2cc4-43c1-985d-a019d3bac8b6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:08:37.710859 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a41bfbf-2cc4-43c1-985d-a019d3bac8b6 admin admin] REQUEST_METHOD: `PUT` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:08:37.711169 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a41bfbf-2cc4-43c1-985d-a019d3bac8b6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:08:37.711447 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a41bfbf-2cc4-43c1-985d-a019d3bac8b6 admin admin] PATH_INFO: `/v3/projects/2cf92f46cf6b41dabd418a33954b35ba/users/b72d1e75efc642acaa87c74a45e7ada8/roles/031d4e1a23e34f71858fc4a6dfbd4ccb` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:08:37.713421 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a41bfbf-2cc4-43c1-985d-a019d3bac8b6 admin admin] RBAC: Authorizing `identity:create_grant(project_id=2cf92f46cf6b41dabd418a33954b35ba, user_id=b72d1e75efc642acaa87c74a45e7ada8, role_id=031d4e1a23e34f71858fc4a6dfbd4ccb)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:08:37.714450 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8aaa486-6781-405c-b865-a82c2c2bfda7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:08:37.715114 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8aaa486-6781-405c-b865-a82c2c2bfda7 admin admin] REQUEST_METHOD: `PUT` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:08:37.715396 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8aaa486-6781-405c-b865-a82c2c2bfda7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:08:37.715672 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8aaa486-6781-405c-b865-a82c2c2bfda7 admin admin] PATH_INFO: `/v3/projects/8fab8596323c484f806904158532bb49/users/77c7141135ae4b8fb3c4cbc3fa153687/roles/15db4aaca3264ae29bde38903ba43471` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:08:37.716186 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a41bfbf-2cc4-43c1-985d-a019d3bac8b6 admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:08:37.728393 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1100/2159] 10.222.0.50 () {60 vars in 1483 bytes} [Tue Sep 13 16:08:37 2022] PUT /identity/v3/projects/2cf92f46cf6b41dabd418a33954b35ba/users/b72d1e75efc642acaa87c74a45e7ada8/roles/031d4e1a23e34f71858fc4a6dfbd4ccb => generated 0 bytes in 34 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 13 16:08:37.730944 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8aaa486-6781-405c-b865-a82c2c2bfda7 admin admin] RBAC: Authorizing `identity:create_grant(project_id=8fab8596323c484f806904158532bb49, user_id=77c7141135ae4b8fb3c4cbc3fa153687, role_id=15db4aaca3264ae29bde38903ba43471)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:08:37.733594 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8aaa486-6781-405c-b865-a82c2c2bfda7 admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:08:37.740422 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35ebdfae-4901-4220-89e2-61638b18a4f8 None None] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:08:37.740789 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35ebdfae-4901-4220-89e2-61638b18a4f8 None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:08:37.741019 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35ebdfae-4901-4220-89e2-61638b18a4f8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:08:37.751387 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1060/2160] 10.222.0.50 () {60 vars in 1483 bytes} [Tue Sep 13 16:08:37 2022] PUT /identity/v3/projects/8fab8596323c484f806904158532bb49/users/77c7141135ae4b8fb3c4cbc3fa153687/roles/15db4aaca3264ae29bde38903ba43471 => generated 0 bytes in 49 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 13 16:08:37.752841 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-217a6442-c287-4459-9d8e-bfb00af74593 None None] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:08:37.753132 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-217a6442-c287-4459-9d8e-bfb00af74593 None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:08:37.753442 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-217a6442-c287-4459-9d8e-bfb00af74593 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:08:37.771152 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.auth.core [None req-35ebdfae-4901-4220-89e2-61638b18a4f8 None None] MFA Rules not processed for user `b72d1e75efc642acaa87c74a45e7ada8`. Rule list: `[]` (Enabled: `True`). {{(pid=81437) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:08:37.772422 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-35ebdfae-4901-4220-89e2-61638b18a4f8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:08:37.772917 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.auth.core [None req-217a6442-c287-4459-9d8e-bfb00af74593 None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81438) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:08:37.774095 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-217a6442-c287-4459-9d8e-bfb00af74593 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:08:37.833782 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1061/2161] 10.222.0.50 () {58 vars in 958 bytes} [Tue Sep 13 16:08:37 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 82 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 13 16:08:37.841149 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1101/2162] 10.222.0.50 () {58 vars in 958 bytes} [Tue Sep 13 16:08:37 2022] POST /identity/v3/auth/tokens => generated 3323 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 13 16:08:37.843739 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-217a6442-c287-4459-9d8e-bfb00af74593 None None] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:08:37.845218 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-217a6442-c287-4459-9d8e-bfb00af74593 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:08:37.853113 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35ebdfae-4901-4220-89e2-61638b18a4f8 None None] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:08:37.854473 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-35ebdfae-4901-4220-89e2-61638b18a4f8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:08:37.879492 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:08:37.910428 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-217a6442-c287-4459-9d8e-bfb00af74593 None None] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:08:37.912003 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35ebdfae-4901-4220-89e2-61638b18a4f8 None None] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:08:37.914122 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f0f3634-c82b-49fa-8779-a0775bbfa1a8 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f436f07939d4855a21b195bc7566244', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:08:37.914728 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f0f3634-c82b-49fa-8779-a0775bbfa1a8 service neutron] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:08:37.915038 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f0f3634-c82b-49fa-8779-a0775bbfa1a8 service neutron] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:08:37.915310 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f0f3634-c82b-49fa-8779-a0775bbfa1a8 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:08:37.915927 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d643aa8-f595-45b3-aea0-f6adb3239b8f service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7fd6b5afd67b48f9b0f0597ccd4cf4df', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:08:37.916536 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d643aa8-f595-45b3-aea0-f6adb3239b8f service cinder] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:08:37.916818 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d643aa8-f595-45b3-aea0-f6adb3239b8f service cinder] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:08:37.917095 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d643aa8-f595-45b3-aea0-f6adb3239b8f service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:08:37.924490 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f0f3634-c82b-49fa-8779-a0775bbfa1a8 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:08:37.926752 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f0f3634-c82b-49fa-8779-a0775bbfa1a8 service neutron] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:08:37.927752 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d643aa8-f595-45b3-aea0-f6adb3239b8f service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:08:37.929307 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d643aa8-f595-45b3-aea0-f6adb3239b8f service cinder] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:08:37.936542 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1062/2163] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 16:08:37 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 93 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 16:08:37.940568 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1102/2164] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 16:08:37 2022] GET /identity/v3/auth/tokens => generated 3323 bytes in 88 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 13 16:08:38.341446 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f0f3634-c82b-49fa-8779-a0775bbfa1a8 service neutron] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:08:38.352412 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f0f3634-c82b-49fa-8779-a0775bbfa1a8 service neutron] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:08:38.357145 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c239a25-1706-4c72-a99c-325d1a374a0a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:08:38.357765 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c239a25-1706-4c72-a99c-325d1a374a0a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:08:38.358044 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c239a25-1706-4c72-a99c-325d1a374a0a admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:08:38.358317 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c239a25-1706-4c72-a99c-325d1a374a0a admin admin] PATH_INFO: `/v3/users/14f60a07cff34e8896d2ecb6c14a964e` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:08:38.359576 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c239a25-1706-4c72-a99c-325d1a374a0a admin admin] RBAC: Authorizing `identity:delete_user(user_id=14f60a07cff34e8896d2ecb6c14a964e)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:08:38.361685 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c239a25-1706-4c72-a99c-325d1a374a0a admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:08:38.416309 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-2c239a25-1706-4c72-a99c-325d1a374a0a admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '14f60a07cff34e8896d2ecb6c14a964e'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:08:38.433261 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-2c239a25-1706-4c72-a99c-325d1a374a0a admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '14f60a07cff34e8896d2ecb6c14a964e'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:08:38.449287 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-2c239a25-1706-4c72-a99c-325d1a374a0a admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '14f60a07cff34e8896d2ecb6c14a964e'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:08:38.450018 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-2c239a25-1706-4c72-a99c-325d1a374a0a admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '14f60a07cff34e8896d2ecb6c14a964e'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:08:38.458131 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-2c239a25-1706-4c72-a99c-325d1a374a0a admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '14f60a07cff34e8896d2ecb6c14a964e'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:08:38.464405 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1063/2165] 10.222.0.50 () {58 vars in 1224 bytes} [Tue Sep 13 16:08:38 2022] DELETE /identity/v3/users/14f60a07cff34e8896d2ecb6c14a964e => generated 0 bytes in 124 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 16:08:38.469104 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7245a09-406a-41c2-90bc-dc0d376a9736 None None] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:08:38.469454 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7245a09-406a-41c2-90bc-dc0d376a9736 None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:08:38.469795 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7245a09-406a-41c2-90bc-dc0d376a9736 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:08:38.489090 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.auth.core [None req-a7245a09-406a-41c2-90bc-dc0d376a9736 None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81437) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:08:38.490398 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-a7245a09-406a-41c2-90bc-dc0d376a9736 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:08:38.503077 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:08:38.564493 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1103/2166] 10.222.0.50 () {58 vars in 958 bytes} [Tue Sep 13 16:08:38 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:08:38.572870 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c239a25-1706-4c72-a99c-325d1a374a0a admin admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:08:38.576050 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-2c239a25-1706-4c72-a99c-325d1a374a0a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:08:38.645276 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c239a25-1706-4c72-a99c-325d1a374a0a admin admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:08:38.648732 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c3bad1b-7c68-44e9-9501-eca571dbdaba service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f436f07939d4855a21b195bc7566244', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:08:38.649369 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c3bad1b-7c68-44e9-9501-eca571dbdaba service neutron] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:08:38.649643 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c3bad1b-7c68-44e9-9501-eca571dbdaba service neutron] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:08:38.650325 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c3bad1b-7c68-44e9-9501-eca571dbdaba service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:08:38.662062 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c3bad1b-7c68-44e9-9501-eca571dbdaba service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:08:38.663657 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c3bad1b-7c68-44e9-9501-eca571dbdaba service neutron] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:08:38.674203 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1064/2167] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 16:08:38 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 103 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 16:08:39.133736 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7245a09-406a-41c2-90bc-dc0d376a9736 None None] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:08:39.135332 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-a7245a09-406a-41c2-90bc-dc0d376a9736 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:08:39.151197 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7245a09-406a-41c2-90bc-dc0d376a9736 None None] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:08:39.157256 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-812e17e1-bb2b-4179-b29d-98fde0f69807 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:08:39.157906 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-812e17e1-bb2b-4179-b29d-98fde0f69807 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:08:39.158183 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-812e17e1-bb2b-4179-b29d-98fde0f69807 admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:08:39.158586 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-812e17e1-bb2b-4179-b29d-98fde0f69807 admin admin] PATH_INFO: `/v3/projects/cdf8f2889bcf49b88b67c7ab6d872a9e` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:08:39.160334 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-812e17e1-bb2b-4179-b29d-98fde0f69807 admin admin] RBAC: Authorizing `identity:delete_project(project_id=cdf8f2889bcf49b88b67c7ab6d872a9e)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:08:39.163291 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-812e17e1-bb2b-4179-b29d-98fde0f69807 admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:08:39.234766 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-812e17e1-bb2b-4179-b29d-98fde0f69807 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'cdf8f2889bcf49b88b67c7ab6d872a9e', 'request_id': 'req-812e17e1-bb2b-4179-b29d-98fde0f69807'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:08:39.248084 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-812e17e1-bb2b-4179-b29d-98fde0f69807 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'cdf8f2889bcf49b88b67c7ab6d872a9e', 'request_id': 'req-812e17e1-bb2b-4179-b29d-98fde0f69807'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:08:39.257516 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-812e17e1-bb2b-4179-b29d-98fde0f69807 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'cdf8f2889bcf49b88b67c7ab6d872a9e', 'request_id': 'req-812e17e1-bb2b-4179-b29d-98fde0f69807'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:08:39.271035 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-812e17e1-bb2b-4179-b29d-98fde0f69807 admin admin] The token cache is being invalidate because project cdf8f2889bcf49b88b67c7ab6d872a9e was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81437) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 13 16:08:39.272579 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-812e17e1-bb2b-4179-b29d-98fde0f69807 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:08:39.273539 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-812e17e1-bb2b-4179-b29d-98fde0f69807 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:08:39.275566 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1104/2168] 10.222.0.50 () {58 vars in 1233 bytes} [Tue Sep 13 16:08:39 2022] DELETE /identity/v3/projects/cdf8f2889bcf49b88b67c7ab6d872a9e => generated 0 bytes in 143 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 16:08:39.279277 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c3bad1b-7c68-44e9-9501-eca571dbdaba service neutron] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:08:39.292738 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c3bad1b-7c68-44e9-9501-eca571dbdaba service neutron] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:08:39.356763 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:08:39.376384 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd6f5dc8-85fb-47a1-8bea-948d9d14fd50 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:08:39.377219 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd6f5dc8-85fb-47a1-8bea-948d9d14fd50 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:08:39.377480 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd6f5dc8-85fb-47a1-8bea-948d9d14fd50 admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:08:39.377755 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd6f5dc8-85fb-47a1-8bea-948d9d14fd50 admin admin] PATH_INFO: `/v3/users/b72d1e75efc642acaa87c74a45e7ada8` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:08:39.379065 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd6f5dc8-85fb-47a1-8bea-948d9d14fd50 admin admin] RBAC: Authorizing `identity:delete_user(user_id=b72d1e75efc642acaa87c74a45e7ada8)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:08:39.381339 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd6f5dc8-85fb-47a1-8bea-948d9d14fd50 admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:08:39.443203 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-cd6f5dc8-85fb-47a1-8bea-948d9d14fd50 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'b72d1e75efc642acaa87c74a45e7ada8'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:08:39.469064 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-cd6f5dc8-85fb-47a1-8bea-948d9d14fd50 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'b72d1e75efc642acaa87c74a45e7ada8'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:08:39.491861 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-cd6f5dc8-85fb-47a1-8bea-948d9d14fd50 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'b72d1e75efc642acaa87c74a45e7ada8'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:08:39.493111 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-cd6f5dc8-85fb-47a1-8bea-948d9d14fd50 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'b72d1e75efc642acaa87c74a45e7ada8'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:08:39.504611 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-cd6f5dc8-85fb-47a1-8bea-948d9d14fd50 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'b72d1e75efc642acaa87c74a45e7ada8'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:08:39.517322 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1065/2169] 10.222.0.50 () {58 vars in 1224 bytes} [Tue Sep 13 16:08:39 2022] DELETE /identity/v3/users/b72d1e75efc642acaa87c74a45e7ada8 => generated 0 bytes in 235 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 16:08:40.230081 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-812e17e1-bb2b-4179-b29d-98fde0f69807 admin admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:08:40.232372 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-812e17e1-bb2b-4179-b29d-98fde0f69807 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:08:40.266309 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-812e17e1-bb2b-4179-b29d-98fde0f69807 admin admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:08:40.340400 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8740048b-fa71-4db8-bd4f-04e12358a0a3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:08:40.341133 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8740048b-fa71-4db8-bd4f-04e12358a0a3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:08:40.341482 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8740048b-fa71-4db8-bd4f-04e12358a0a3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:08:40.341827 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8740048b-fa71-4db8-bd4f-04e12358a0a3 admin admin] PATH_INFO: `/v3/projects/2cf92f46cf6b41dabd418a33954b35ba` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:08:40.343401 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8740048b-fa71-4db8-bd4f-04e12358a0a3 admin admin] RBAC: Authorizing `identity:delete_project(project_id=2cf92f46cf6b41dabd418a33954b35ba)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:08:40.345399 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8740048b-fa71-4db8-bd4f-04e12358a0a3 admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:08:40.390773 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:08:40.396429 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-8740048b-fa71-4db8-bd4f-04e12358a0a3 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '2cf92f46cf6b41dabd418a33954b35ba', 'request_id': 'req-8740048b-fa71-4db8-bd4f-04e12358a0a3'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:08:40.405434 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-8740048b-fa71-4db8-bd4f-04e12358a0a3 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '2cf92f46cf6b41dabd418a33954b35ba', 'request_id': 'req-8740048b-fa71-4db8-bd4f-04e12358a0a3'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:08:40.413199 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-8740048b-fa71-4db8-bd4f-04e12358a0a3 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '2cf92f46cf6b41dabd418a33954b35ba', 'request_id': 'req-8740048b-fa71-4db8-bd4f-04e12358a0a3'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:08:40.423824 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-8740048b-fa71-4db8-bd4f-04e12358a0a3 admin admin] The token cache is being invalidate because project 2cf92f46cf6b41dabd418a33954b35ba was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81437) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 13 16:08:40.424114 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-8740048b-fa71-4db8-bd4f-04e12358a0a3 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:08:40.424761 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-8740048b-fa71-4db8-bd4f-04e12358a0a3 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:08:40.426503 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1105/2170] 10.222.0.50 () {58 vars in 1233 bytes} [Tue Sep 13 16:08:40 2022] DELETE /identity/v3/projects/2cf92f46cf6b41dabd418a33954b35ba => generated 0 bytes in 197 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 16:08:40.432765 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cecc6ec-6111-4b0d-b2dd-18f581d0d412 None None] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:08:40.433250 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cecc6ec-6111-4b0d-b2dd-18f581d0d412 None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:08:40.433340 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cecc6ec-6111-4b0d-b2dd-18f581d0d412 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:08:40.454528 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.auth.core [None req-1cecc6ec-6111-4b0d-b2dd-18f581d0d412 None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81438) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:08:40.455775 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-1cecc6ec-6111-4b0d-b2dd-18f581d0d412 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:08:40.519196 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1066/2171] 10.222.0.50 () {58 vars in 958 bytes} [Tue Sep 13 16:08:40 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:08:40.526748 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75a4810a-3149-40c7-b37e-e45d8229de8f None None] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:08:40.527084 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75a4810a-3149-40c7-b37e-e45d8229de8f None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:08:40.527365 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75a4810a-3149-40c7-b37e-e45d8229de8f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:08:40.547952 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.auth.core [None req-75a4810a-3149-40c7-b37e-e45d8229de8f None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81437) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:08:40.549248 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-75a4810a-3149-40c7-b37e-e45d8229de8f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:08:40.565196 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1106/2172] 10.222.0.50 () {58 vars in 958 bytes} [Tue Sep 13 16:08:40 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 39 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:08:40.570449 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cecc6ec-6111-4b0d-b2dd-18f581d0d412 None None] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:08:40.583372 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cecc6ec-6111-4b0d-b2dd-18f581d0d412 None None] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:08:40.589143 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-878f2189-c909-4e33-9a85-579514632a19 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:08:40.589822 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-878f2189-c909-4e33-9a85-579514632a19 admin admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:08:40.590191 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-878f2189-c909-4e33-9a85-579514632a19 admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:08:40.590492 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-878f2189-c909-4e33-9a85-579514632a19 admin admin] PATH_INFO: `/v3/domains` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:08:40.591313 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-878f2189-c909-4e33-9a85-579514632a19 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81438) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 16:08:40.591622 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-878f2189-c909-4e33-9a85-579514632a19 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:08:40.593361 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-878f2189-c909-4e33-9a85-579514632a19 admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:08:40.607893 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1067/2173] 10.222.0.50 () {58 vars in 1153 bytes} [Tue Sep 13 16:08:40 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 16:08:40.612968 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75a4810a-3149-40c7-b37e-e45d8229de8f None None] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:08:40.623474 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75a4810a-3149-40c7-b37e-e45d8229de8f None None] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:08:40.628995 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f8d8787-f84d-4b4c-9859-8999f74e49b6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:08:40.629877 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f8d8787-f84d-4b4c-9859-8999f74e49b6 admin admin] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:08:40.630156 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f8d8787-f84d-4b4c-9859-8999f74e49b6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:08:40.630431 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f8d8787-f84d-4b4c-9859-8999f74e49b6 admin admin] PATH_INFO: `/v3/projects` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:08:40.631138 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f8d8787-f84d-4b4c-9859-8999f74e49b6 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:08:40.633361 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f8d8787-f84d-4b4c-9859-8999f74e49b6 admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:08:40.722087 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1107/2174] 10.222.0.50 () {60 vars in 1153 bytes} [Tue Sep 13 16:08:40 2022] POST /identity/v3/projects => generated 383 bytes in 110 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 16:08:40.724183 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-878f2189-c909-4e33-9a85-579514632a19 admin admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:08:40.762313 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-878f2189-c909-4e33-9a85-579514632a19 admin admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:08:40.906042 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:08:40.949142 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49b20f38-5850-4645-9757-7ac08ab7265a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:08:40.949777 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49b20f38-5850-4645-9757-7ac08ab7265a admin admin] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:08:40.950089 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49b20f38-5850-4645-9757-7ac08ab7265a admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:08:40.950365 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49b20f38-5850-4645-9757-7ac08ab7265a admin admin] PATH_INFO: `/v3/users` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:08:40.951066 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49b20f38-5850-4645-9757-7ac08ab7265a admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:08:40.953389 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49b20f38-5850-4645-9757-7ac08ab7265a admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:08:41.012627 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1068/2175] 10.222.0.50 () {60 vars in 1144 bytes} [Tue Sep 13 16:08:40 2022] POST /identity/v3/users => generated 292 bytes in 289 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 13 16:08:41.013896 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f8d8787-f84d-4b4c-9859-8999f74e49b6 admin admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:08:41.030721 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f8d8787-f84d-4b4c-9859-8999f74e49b6 admin admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:08:41.048131 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf02b707-9372-494f-8de2-597cab416f2c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:08:41.048701 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf02b707-9372-494f-8de2-597cab416f2c admin admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:08:41.049209 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf02b707-9372-494f-8de2-597cab416f2c admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:08:41.049503 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf02b707-9372-494f-8de2-597cab416f2c admin admin] PATH_INFO: `/v3/roles` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:08:41.050251 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf02b707-9372-494f-8de2-597cab416f2c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:08:41.053995 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf02b707-9372-494f-8de2-597cab416f2c admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:08:41.113226 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49b20f38-5850-4645-9757-7ac08ab7265a admin admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:08:41.119152 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1108/2176] 10.222.0.50 () {58 vars in 1122 bytes} [Tue Sep 13 16:08:41 2022] GET /identity/v3/roles => generated 1420 bytes in 106 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 16:08:41.134775 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49b20f38-5850-4645-9757-7ac08ab7265a admin admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:08:41.149112 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a4ad149-c7eb-4891-8a05-a8c5f30554bb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:08:41.149823 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a4ad149-c7eb-4891-8a05-a8c5f30554bb admin admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:08:41.150175 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a4ad149-c7eb-4891-8a05-a8c5f30554bb admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:08:41.150524 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a4ad149-c7eb-4891-8a05-a8c5f30554bb admin admin] PATH_INFO: `/v3/roles` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:08:41.151271 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a4ad149-c7eb-4891-8a05-a8c5f30554bb admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:08:41.155007 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a4ad149-c7eb-4891-8a05-a8c5f30554bb admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:08:41.175682 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1069/2177] 10.222.0.50 () {58 vars in 1122 bytes} [Tue Sep 13 16:08:41 2022] GET /identity/v3/roles => generated 1420 bytes in 63 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 16:08:41.176864 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf02b707-9372-494f-8de2-597cab416f2c admin admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:08:41.188600 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf02b707-9372-494f-8de2-597cab416f2c admin admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:08:41.200804 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-914d153c-4eda-43eb-ba18-61d09b609d45 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:08:41.201743 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-914d153c-4eda-43eb-ba18-61d09b609d45 admin admin] REQUEST_METHOD: `PUT` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:08:41.202151 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-914d153c-4eda-43eb-ba18-61d09b609d45 admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:08:41.202511 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-914d153c-4eda-43eb-ba18-61d09b609d45 admin admin] PATH_INFO: `/v3/projects/b30c55812b704372a65bfd70422d0cff/users/1303fb8c7b774eea88240f942c0c4bee/roles/15db4aaca3264ae29bde38903ba43471` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:08:41.223009 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-914d153c-4eda-43eb-ba18-61d09b609d45 admin admin] RBAC: Authorizing `identity:create_grant(project_id=b30c55812b704372a65bfd70422d0cff, user_id=1303fb8c7b774eea88240f942c0c4bee, role_id=15db4aaca3264ae29bde38903ba43471)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:08:41.226442 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-914d153c-4eda-43eb-ba18-61d09b609d45 admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:08:41.244376 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1109/2178] 10.222.0.50 () {60 vars in 1483 bytes} [Tue Sep 13 16:08:41 2022] PUT /identity/v3/projects/b30c55812b704372a65bfd70422d0cff/users/1303fb8c7b774eea88240f942c0c4bee/roles/15db4aaca3264ae29bde38903ba43471 => generated 0 bytes in 68 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 16:08:41.255347 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94b409a4-68e8-4001-a609-7e7f5c20c747 None None] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:08:41.255766 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94b409a4-68e8-4001-a609-7e7f5c20c747 None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:08:41.256148 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94b409a4-68e8-4001-a609-7e7f5c20c747 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:08:41.311624 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.auth.core [None req-94b409a4-68e8-4001-a609-7e7f5c20c747 None None] MFA Rules not processed for user `1303fb8c7b774eea88240f942c0c4bee`. Rule list: `[]` (Enabled: `True`). {{(pid=81438) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:08:41.314749 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-94b409a4-68e8-4001-a609-7e7f5c20c747 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:08:41.435226 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1070/2179] 10.222.0.50 () {58 vars in 958 bytes} [Tue Sep 13 16:08:41 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 181 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 13 16:08:41.439409 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-914d153c-4eda-43eb-ba18-61d09b609d45 admin admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:08:41.545988 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-914d153c-4eda-43eb-ba18-61d09b609d45 admin admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:08:41.551519 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66908460-f6ef-45df-9433-060b5c7885f5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:08:41.552808 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66908460-f6ef-45df-9433-060b5c7885f5 admin admin] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:08:41.552808 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66908460-f6ef-45df-9433-060b5c7885f5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:08:41.552808 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66908460-f6ef-45df-9433-060b5c7885f5 admin admin] PATH_INFO: `/v3/projects` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:08:41.553165 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66908460-f6ef-45df-9433-060b5c7885f5 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:08:41.564865 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66908460-f6ef-45df-9433-060b5c7885f5 admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:08:41.611771 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1110/2180] 10.222.0.50 () {60 vars in 1153 bytes} [Tue Sep 13 16:08:41 2022] POST /identity/v3/projects => generated 383 bytes in 172 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 13 16:08:41.611943 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94b409a4-68e8-4001-a609-7e7f5c20c747 None None] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:08:41.625416 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94b409a4-68e8-4001-a609-7e7f5c20c747 None None] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:08:41.711648 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31d15909-cf39-4fbd-9f22-3cf88cf1f0f7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:08:41.712211 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31d15909-cf39-4fbd-9f22-3cf88cf1f0f7 admin admin] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:08:41.712496 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31d15909-cf39-4fbd-9f22-3cf88cf1f0f7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:08:41.712766 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31d15909-cf39-4fbd-9f22-3cf88cf1f0f7 admin admin] PATH_INFO: `/v3/users` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:08:41.713401 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31d15909-cf39-4fbd-9f22-3cf88cf1f0f7 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:08:41.715558 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31d15909-cf39-4fbd-9f22-3cf88cf1f0f7 admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:08:41.740896 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1071/2181] 10.222.0.50 () {60 vars in 1144 bytes} [Tue Sep 13 16:08:41 2022] POST /identity/v3/users => generated 298 bytes in 130 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 13 16:08:41.745085 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66908460-f6ef-45df-9433-060b5c7885f5 admin admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:08:41.760537 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66908460-f6ef-45df-9433-060b5c7885f5 admin admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:08:41.767120 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9882da9f-daee-468f-8815-a10a407c9ad1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:08:41.767793 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9882da9f-daee-468f-8815-a10a407c9ad1 admin admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:08:41.769309 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9882da9f-daee-468f-8815-a10a407c9ad1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:08:41.769813 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9882da9f-daee-468f-8815-a10a407c9ad1 admin admin] PATH_INFO: `/v3/roles` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:08:41.770648 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9882da9f-daee-468f-8815-a10a407c9ad1 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:08:41.773451 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9882da9f-daee-468f-8815-a10a407c9ad1 admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:08:41.788337 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1111/2182] 10.222.0.50 () {58 vars in 1122 bytes} [Tue Sep 13 16:08:41 2022] GET /identity/v3/roles => generated 1420 bytes in 44 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 16:08:41.792695 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31d15909-cf39-4fbd-9f22-3cf88cf1f0f7 admin admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:08:41.804538 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31d15909-cf39-4fbd-9f22-3cf88cf1f0f7 admin admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:08:41.809886 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2513c636-ff31-498f-968e-8cdb0fd01c7f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:08:41.810712 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2513c636-ff31-498f-968e-8cdb0fd01c7f admin admin] REQUEST_METHOD: `PUT` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:08:41.811027 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2513c636-ff31-498f-968e-8cdb0fd01c7f admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:08:41.811302 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2513c636-ff31-498f-968e-8cdb0fd01c7f admin admin] PATH_INFO: `/v3/projects/a03553ba854c438fbd0d02b5c608d975/users/a82bb8e5858a4f49a95a8f01bdf76019/roles/031d4e1a23e34f71858fc4a6dfbd4ccb` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:08:41.829414 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2513c636-ff31-498f-968e-8cdb0fd01c7f admin admin] RBAC: Authorizing `identity:create_grant(project_id=a03553ba854c438fbd0d02b5c608d975, user_id=a82bb8e5858a4f49a95a8f01bdf76019, role_id=031d4e1a23e34f71858fc4a6dfbd4ccb)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:08:41.832686 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2513c636-ff31-498f-968e-8cdb0fd01c7f admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:08:41.849866 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1072/2183] 10.222.0.50 () {60 vars in 1483 bytes} [Tue Sep 13 16:08:41 2022] PUT /identity/v3/projects/a03553ba854c438fbd0d02b5c608d975/users/a82bb8e5858a4f49a95a8f01bdf76019/roles/031d4e1a23e34f71858fc4a6dfbd4ccb => generated 0 bytes in 58 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 16:08:41.853762 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9882da9f-daee-468f-8815-a10a407c9ad1 admin admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:08:41.868415 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9882da9f-daee-468f-8815-a10a407c9ad1 admin admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:08:41.922229 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=92,keystone:DELETE=4,keystone:INSERT=4 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:08:41.948954 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01a8acaf-e1ea-4bea-8052-74f60615aa5e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:08:41.949662 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01a8acaf-e1ea-4bea-8052-74f60615aa5e admin admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:08:41.950018 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01a8acaf-e1ea-4bea-8052-74f60615aa5e admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:08:41.950812 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01a8acaf-e1ea-4bea-8052-74f60615aa5e admin admin] PATH_INFO: `/v3/roles` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:08:41.951137 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01a8acaf-e1ea-4bea-8052-74f60615aa5e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:08:41.952754 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01a8acaf-e1ea-4bea-8052-74f60615aa5e admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:08:41.967015 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1112/2184] 10.222.0.50 () {58 vars in 1122 bytes} [Tue Sep 13 16:08:41 2022] GET /identity/v3/roles => generated 1420 bytes in 112 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 16:08:41.969728 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2513c636-ff31-498f-968e-8cdb0fd01c7f admin admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:08:41.981171 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2513c636-ff31-498f-968e-8cdb0fd01c7f admin admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:08:41.985889 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38c16d59-a0b2-4970-8e28-feda3605b529 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:08:41.986615 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38c16d59-a0b2-4970-8e28-feda3605b529 admin admin] REQUEST_METHOD: `PUT` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:08:41.987004 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38c16d59-a0b2-4970-8e28-feda3605b529 admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:08:41.988021 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38c16d59-a0b2-4970-8e28-feda3605b529 admin admin] PATH_INFO: `/v3/projects/a03553ba854c438fbd0d02b5c608d975/users/a82bb8e5858a4f49a95a8f01bdf76019/roles/031d4e1a23e34f71858fc4a6dfbd4ccb` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:08:41.989862 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38c16d59-a0b2-4970-8e28-feda3605b529 admin admin] RBAC: Authorizing `identity:create_grant(project_id=a03553ba854c438fbd0d02b5c608d975, user_id=a82bb8e5858a4f49a95a8f01bdf76019, role_id=031d4e1a23e34f71858fc4a6dfbd4ccb)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:08:41.992690 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38c16d59-a0b2-4970-8e28-feda3605b529 admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:08:42.006128 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1073/2185] 10.222.0.50 () {60 vars in 1483 bytes} [Tue Sep 13 16:08:41 2022] PUT /identity/v3/projects/a03553ba854c438fbd0d02b5c608d975/users/a82bb8e5858a4f49a95a8f01bdf76019/roles/031d4e1a23e34f71858fc4a6dfbd4ccb => generated 0 bytes in 37 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 16:08:42.016951 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da0d0945-7fda-429f-aa47-9c49dea9ad29 None None] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:08:42.017303 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da0d0945-7fda-429f-aa47-9c49dea9ad29 None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:08:42.017647 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da0d0945-7fda-429f-aa47-9c49dea9ad29 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:08:42.050288 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.auth.core [None req-da0d0945-7fda-429f-aa47-9c49dea9ad29 None None] MFA Rules not processed for user `a82bb8e5858a4f49a95a8f01bdf76019`. Rule list: `[]` (Enabled: `True`). {{(pid=81437) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:08:42.051873 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-da0d0945-7fda-429f-aa47-9c49dea9ad29 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:08:42.142973 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1113/2186] 10.222.0.50 () {58 vars in 958 bytes} [Tue Sep 13 16:08:42 2022] POST /identity/v3/auth/tokens => generated 3327 bytes in 127 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:08:42.155496 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38c16d59-a0b2-4970-8e28-feda3605b529 admin admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:08:42.158699 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-38c16d59-a0b2-4970-8e28-feda3605b529 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:08:42.177674 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=92,keystone:INSERT=8 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:08:42.258002 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38c16d59-a0b2-4970-8e28-feda3605b529 admin admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:08:42.261922 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ebf3d20-533f-4f76-a389-0ea3ea5c7cb5 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7fd6b5afd67b48f9b0f0597ccd4cf4df', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:08:42.262527 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ebf3d20-533f-4f76-a389-0ea3ea5c7cb5 service cinder] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:08:42.263333 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ebf3d20-533f-4f76-a389-0ea3ea5c7cb5 service cinder] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:08:42.263333 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ebf3d20-533f-4f76-a389-0ea3ea5c7cb5 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:08:42.351782 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ebf3d20-533f-4f76-a389-0ea3ea5c7cb5 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:08:42.355368 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ebf3d20-533f-4f76-a389-0ea3ea5c7cb5 service cinder] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:08:42.375875 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1074/2187] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 16:08:42 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 221 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 16:08:43.628796 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c68f61d-8604-4e4e-ab27-ba9e7af12f7b None None] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:08:43.629638 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c68f61d-8604-4e4e-ab27-ba9e7af12f7b None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:08:43.630134 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c68f61d-8604-4e4e-ab27-ba9e7af12f7b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:08:43.667950 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.auth.core [None req-5c68f61d-8604-4e4e-ab27-ba9e7af12f7b None None] MFA Rules not processed for user `77c7141135ae4b8fb3c4cbc3fa153687`. Rule list: `[]` (Enabled: `True`). {{(pid=81437) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:08:43.670336 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-5c68f61d-8604-4e4e-ab27-ba9e7af12f7b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:08:43.769367 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1114/2188] 10.222.0.50 () {58 vars in 958 bytes} [Tue Sep 13 16:08:43 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 145 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:08:43.794635 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ebf3d20-533f-4f76-a389-0ea3ea5c7cb5 service cinder] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:08:43.799523 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-4ebf3d20-533f-4f76-a389-0ea3ea5c7cb5 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:08:43.927420 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ebf3d20-533f-4f76-a389-0ea3ea5c7cb5 service cinder] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:08:43.935639 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee4d1f25-2538-47bb-ac60-f58992314a40 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4e039b46c5e3445bbb39904b00c5430e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'admin', 'ResellerAdmin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:08:43.937055 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee4d1f25-2538-47bb-ac60-f58992314a40 service nova] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:08:43.937411 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee4d1f25-2538-47bb-ac60-f58992314a40 service nova] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:08:43.937640 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee4d1f25-2538-47bb-ac60-f58992314a40 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:08:43.949167 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee4d1f25-2538-47bb-ac60-f58992314a40 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:08:43.950883 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee4d1f25-2538-47bb-ac60-f58992314a40 service nova] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:08:43.964856 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1075/2189] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 16:08:43 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 171 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 16:08:45.234851 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c68f61d-8604-4e4e-ab27-ba9e7af12f7b None None] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:08:45.236346 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-5c68f61d-8604-4e4e-ab27-ba9e7af12f7b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:08:45.318132 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:08:45.340552 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c68f61d-8604-4e4e-ab27-ba9e7af12f7b None None] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:08:45.361616 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78bae381-9dfa-48a8-9b79-9dfa2450e1b0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:08:45.364420 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78bae381-9dfa-48a8-9b79-9dfa2450e1b0 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:08:45.364784 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78bae381-9dfa-48a8-9b79-9dfa2450e1b0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:08:45.365132 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78bae381-9dfa-48a8-9b79-9dfa2450e1b0 admin admin] PATH_INFO: `/v3/users/2092fb6d24d040419b1ea48ea4945831` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:08:45.367104 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78bae381-9dfa-48a8-9b79-9dfa2450e1b0 admin admin] RBAC: Authorizing `identity:delete_user(user_id=2092fb6d24d040419b1ea48ea4945831)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:08:45.370723 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78bae381-9dfa-48a8-9b79-9dfa2450e1b0 admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:08:45.453476 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-78bae381-9dfa-48a8-9b79-9dfa2450e1b0 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '2092fb6d24d040419b1ea48ea4945831'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:08:45.482651 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-78bae381-9dfa-48a8-9b79-9dfa2450e1b0 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '2092fb6d24d040419b1ea48ea4945831'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:08:45.534780 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-78bae381-9dfa-48a8-9b79-9dfa2450e1b0 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '2092fb6d24d040419b1ea48ea4945831'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:08:45.542600 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-78bae381-9dfa-48a8-9b79-9dfa2450e1b0 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '2092fb6d24d040419b1ea48ea4945831'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:08:45.571322 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-78bae381-9dfa-48a8-9b79-9dfa2450e1b0 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '2092fb6d24d040419b1ea48ea4945831'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:08:45.595846 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1115/2190] 10.222.0.50 () {58 vars in 1224 bytes} [Tue Sep 13 16:08:45 2022] DELETE /identity/v3/users/2092fb6d24d040419b1ea48ea4945831 => generated 0 bytes in 359 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 16:08:45.598777 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd7ddb19-917a-43a0-8810-687f0d16dd97 None None] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:08:45.599387 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd7ddb19-917a-43a0-8810-687f0d16dd97 None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:08:45.599387 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd7ddb19-917a-43a0-8810-687f0d16dd97 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:08:45.624308 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-457c20ce-749b-461b-b76f-71f818bd9596 None None] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:08:45.624667 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-457c20ce-749b-461b-b76f-71f818bd9596 None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:08:45.625013 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-457c20ce-749b-461b-b76f-71f818bd9596 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:08:45.660448 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.auth.core [None req-dd7ddb19-917a-43a0-8810-687f0d16dd97 None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81438) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:08:45.662158 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-dd7ddb19-917a-43a0-8810-687f0d16dd97 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:08:45.694297 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.auth.core [None req-457c20ce-749b-461b-b76f-71f818bd9596 None None] MFA Rules not processed for user `77c7141135ae4b8fb3c4cbc3fa153687`. Rule list: `[]` (Enabled: `True`). {{(pid=81437) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:08:45.696453 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-457c20ce-749b-461b-b76f-71f818bd9596 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:08:45.774153 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:08:45.806527 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1076/2191] 10.222.0.50 () {58 vars in 958 bytes} [Tue Sep 13 16:08:45 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 209 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:08:45.828817 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1116/2192] 10.222.0.50 () {58 vars in 958 bytes} [Tue Sep 13 16:08:45 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 206 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 13 16:08:45.834786 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd7ddb19-917a-43a0-8810-687f0d16dd97 None None] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:08:45.841563 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-dd7ddb19-917a-43a0-8810-687f0d16dd97 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:08:45.956503 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd7ddb19-917a-43a0-8810-687f0d16dd97 None None] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:08:45.961202 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95878f9f-b9f1-4cc8-9b3b-0c252f4abc55 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f436f07939d4855a21b195bc7566244', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:08:45.962024 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95878f9f-b9f1-4cc8-9b3b-0c252f4abc55 service neutron] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:08:45.962455 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95878f9f-b9f1-4cc8-9b3b-0c252f4abc55 service neutron] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:08:45.963087 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95878f9f-b9f1-4cc8-9b3b-0c252f4abc55 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:08:45.976646 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95878f9f-b9f1-4cc8-9b3b-0c252f4abc55 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:08:45.978460 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95878f9f-b9f1-4cc8-9b3b-0c252f4abc55 service neutron] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:08:45.992407 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1077/2193] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 16:08:45 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 158 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 16:08:46.599734 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-457c20ce-749b-461b-b76f-71f818bd9596 None None] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:08:46.601277 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-457c20ce-749b-461b-b76f-71f818bd9596 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:08:46.620635 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-457c20ce-749b-461b-b76f-71f818bd9596 None None] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:08:46.630024 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c4cdd92-7430-4698-91ba-6ed9e13fe24a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:08:46.631505 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c4cdd92-7430-4698-91ba-6ed9e13fe24a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:08:46.632110 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c4cdd92-7430-4698-91ba-6ed9e13fe24a admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:08:46.632586 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c4cdd92-7430-4698-91ba-6ed9e13fe24a admin admin] PATH_INFO: `/v3/projects/718b3e0e848f4540ba0237b60e092664` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:08:46.634405 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c4cdd92-7430-4698-91ba-6ed9e13fe24a admin admin] RBAC: Authorizing `identity:delete_project(project_id=718b3e0e848f4540ba0237b60e092664)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:08:46.644256 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c4cdd92-7430-4698-91ba-6ed9e13fe24a admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:08:46.713397 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-6c4cdd92-7430-4698-91ba-6ed9e13fe24a admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '718b3e0e848f4540ba0237b60e092664', 'request_id': 'req-6c4cdd92-7430-4698-91ba-6ed9e13fe24a'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:08:46.723731 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-6c4cdd92-7430-4698-91ba-6ed9e13fe24a admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '718b3e0e848f4540ba0237b60e092664', 'request_id': 'req-6c4cdd92-7430-4698-91ba-6ed9e13fe24a'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:08:46.731575 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-6c4cdd92-7430-4698-91ba-6ed9e13fe24a admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '718b3e0e848f4540ba0237b60e092664', 'request_id': 'req-6c4cdd92-7430-4698-91ba-6ed9e13fe24a'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:08:46.750001 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=81,keystone:DELETE=17,keystone:INSERT=2 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:08:46.755950 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-6c4cdd92-7430-4698-91ba-6ed9e13fe24a admin admin] The token cache is being invalidate because project 718b3e0e848f4540ba0237b60e092664 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81437) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 13 16:08:46.756368 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-6c4cdd92-7430-4698-91ba-6ed9e13fe24a admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:08:46.757575 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-6c4cdd92-7430-4698-91ba-6ed9e13fe24a admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:08:46.764625 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1117/2194] 10.222.0.50 () {58 vars in 1233 bytes} [Tue Sep 13 16:08:46 2022] DELETE /identity/v3/projects/718b3e0e848f4540ba0237b60e092664 => generated 0 bytes in 162 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 16:08:46.766618 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a451980-d154-4fba-be2a-4361449d36fa None None] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:08:46.767235 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a451980-d154-4fba-be2a-4361449d36fa None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:08:46.767235 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a451980-d154-4fba-be2a-4361449d36fa None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:08:46.807862 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.auth.core [None req-7a451980-d154-4fba-be2a-4361449d36fa None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81438) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:08:46.809099 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-7a451980-d154-4fba-be2a-4361449d36fa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:08:46.976210 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1078/2195] 10.222.0.50 () {58 vars in 958 bytes} [Tue Sep 13 16:08:46 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 211 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:08:46.983833 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e03acc1f-460a-4872-9b71-67a9ba5b6a53 None None] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:08:46.984378 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e03acc1f-460a-4872-9b71-67a9ba5b6a53 None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:08:46.984835 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e03acc1f-460a-4872-9b71-67a9ba5b6a53 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:08:47.020270 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.auth.core [None req-e03acc1f-460a-4872-9b71-67a9ba5b6a53 None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81437) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:08:47.021985 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-e03acc1f-460a-4872-9b71-67a9ba5b6a53 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:08:47.043737 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1118/2196] 10.222.0.50 () {58 vars in 958 bytes} [Tue Sep 13 16:08:46 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 61 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:08:47.047173 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a451980-d154-4fba-be2a-4361449d36fa None None] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:08:47.059149 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a451980-d154-4fba-be2a-4361449d36fa None None] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:08:47.080141 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a28990d1-98f2-4bdc-ac6f-f6793af83f5b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:08:47.080852 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a28990d1-98f2-4bdc-ac6f-f6793af83f5b admin admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:08:47.081137 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a28990d1-98f2-4bdc-ac6f-f6793af83f5b admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:08:47.081412 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a28990d1-98f2-4bdc-ac6f-f6793af83f5b admin admin] PATH_INFO: `/v3/domains` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:08:47.086047 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a28990d1-98f2-4bdc-ac6f-f6793af83f5b admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81438) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 16:08:47.086364 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a28990d1-98f2-4bdc-ac6f-f6793af83f5b admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:08:47.089560 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a28990d1-98f2-4bdc-ac6f-f6793af83f5b admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:08:47.122938 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1079/2197] 10.222.0.50 () {58 vars in 1153 bytes} [Tue Sep 13 16:08:47 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 74 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 16:08:47.124969 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e03acc1f-460a-4872-9b71-67a9ba5b6a53 None None] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:08:47.136639 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e03acc1f-460a-4872-9b71-67a9ba5b6a53 None None] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:08:47.141570 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81d3f950-4d41-4155-8154-bb6970aa0d46 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:08:47.142208 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81d3f950-4d41-4155-8154-bb6970aa0d46 admin admin] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:08:47.142557 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81d3f950-4d41-4155-8154-bb6970aa0d46 admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:08:47.143198 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81d3f950-4d41-4155-8154-bb6970aa0d46 admin admin] PATH_INFO: `/v3/projects` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:08:47.143922 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81d3f950-4d41-4155-8154-bb6970aa0d46 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:08:47.146209 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81d3f950-4d41-4155-8154-bb6970aa0d46 admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:08:47.181785 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1119/2198] 10.222.0.50 () {60 vars in 1153 bytes} [Tue Sep 13 16:08:47 2022] POST /identity/v3/projects => generated 383 bytes in 57 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 16:08:47.185557 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a28990d1-98f2-4bdc-ac6f-f6793af83f5b admin admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:08:47.200706 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a28990d1-98f2-4bdc-ac6f-f6793af83f5b admin admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:08:47.278697 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a471caf6-ec18-486d-9348-e301ce9906de admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:08:47.279664 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a471caf6-ec18-486d-9348-e301ce9906de admin admin] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:08:47.281265 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a471caf6-ec18-486d-9348-e301ce9906de admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:08:47.281532 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a471caf6-ec18-486d-9348-e301ce9906de admin admin] PATH_INFO: `/v3/users` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:08:47.282280 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a471caf6-ec18-486d-9348-e301ce9906de admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:08:47.284629 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a471caf6-ec18-486d-9348-e301ce9906de admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:08:47.324359 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1080/2199] 10.222.0.50 () {60 vars in 1144 bytes} [Tue Sep 13 16:08:47 2022] POST /identity/v3/users => generated 292 bytes in 139 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 16:08:47.341628 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81d3f950-4d41-4155-8154-bb6970aa0d46 admin admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:08:47.356517 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81d3f950-4d41-4155-8154-bb6970aa0d46 admin admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:08:47.387952 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5bc2be5-47ba-4717-a581-21751b4b8715 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:08:47.388994 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5bc2be5-47ba-4717-a581-21751b4b8715 admin admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:08:47.389421 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5bc2be5-47ba-4717-a581-21751b4b8715 admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:08:47.390030 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5bc2be5-47ba-4717-a581-21751b4b8715 admin admin] PATH_INFO: `/v3/roles` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:08:47.390842 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5bc2be5-47ba-4717-a581-21751b4b8715 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:08:47.392773 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5bc2be5-47ba-4717-a581-21751b4b8715 admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:08:47.406824 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1120/2200] 10.222.0.50 () {58 vars in 1122 bytes} [Tue Sep 13 16:08:47 2022] GET /identity/v3/roles => generated 1420 bytes in 66 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 16:08:47.411809 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a471caf6-ec18-486d-9348-e301ce9906de admin admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:08:47.431996 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a471caf6-ec18-486d-9348-e301ce9906de admin admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:08:47.444532 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c070e264-0aed-4706-9aa3-b7e90a2b9b55 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:08:47.445620 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c070e264-0aed-4706-9aa3-b7e90a2b9b55 admin admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:08:47.445904 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c070e264-0aed-4706-9aa3-b7e90a2b9b55 admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:08:47.446186 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c070e264-0aed-4706-9aa3-b7e90a2b9b55 admin admin] PATH_INFO: `/v3/roles` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:08:47.447540 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c070e264-0aed-4706-9aa3-b7e90a2b9b55 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:08:47.449742 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c070e264-0aed-4706-9aa3-b7e90a2b9b55 admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:08:47.457590 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:08:47.462326 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5bc2be5-47ba-4717-a581-21751b4b8715 admin admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:08:47.464454 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-d5bc2be5-47ba-4717-a581-21751b4b8715 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:08:47.467845 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1081/2201] 10.222.0.50 () {58 vars in 1122 bytes} [Tue Sep 13 16:08:47 2022] GET /identity/v3/roles => generated 1420 bytes in 56 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 16:08:47.476913 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c070e264-0aed-4706-9aa3-b7e90a2b9b55 admin admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:08:47.491763 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c070e264-0aed-4706-9aa3-b7e90a2b9b55 admin admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:08:47.508911 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-616f3f1a-da10-494e-b01f-c98b76837157 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:08:47.509702 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-616f3f1a-da10-494e-b01f-c98b76837157 admin admin] REQUEST_METHOD: `PUT` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:08:47.510058 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-616f3f1a-da10-494e-b01f-c98b76837157 admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:08:47.510661 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-616f3f1a-da10-494e-b01f-c98b76837157 admin admin] PATH_INFO: `/v3/projects/673d53e0757746708c6ca7b9d947b409/users/313ac02befbf44ebabf3b41f84245c1f/roles/15db4aaca3264ae29bde38903ba43471` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:08:47.537621 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-616f3f1a-da10-494e-b01f-c98b76837157 admin admin] RBAC: Authorizing `identity:create_grant(project_id=673d53e0757746708c6ca7b9d947b409, user_id=313ac02befbf44ebabf3b41f84245c1f, role_id=15db4aaca3264ae29bde38903ba43471)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:08:47.540760 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-616f3f1a-da10-494e-b01f-c98b76837157 admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:08:47.557319 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5bc2be5-47ba-4717-a581-21751b4b8715 admin admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:08:47.562295 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1082/2202] 10.222.0.50 () {60 vars in 1483 bytes} [Tue Sep 13 16:08:47 2022] PUT /identity/v3/projects/673d53e0757746708c6ca7b9d947b409/users/313ac02befbf44ebabf3b41f84245c1f/roles/15db4aaca3264ae29bde38903ba43471 => generated 0 bytes in 88 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 16:08:47.571930 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d27fe078-c698-425f-b232-52867f4e3fc6 None None] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:08:47.572519 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d27fe078-c698-425f-b232-52867f4e3fc6 None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:08:47.572972 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d27fe078-c698-425f-b232-52867f4e3fc6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:08:47.601158 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.auth.core [None req-d27fe078-c698-425f-b232-52867f4e3fc6 None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81438) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:08:47.603292 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-d27fe078-c698-425f-b232-52867f4e3fc6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:08:47.614954 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d7e8975-f5b2-4a3c-8a32-f46eede0af49 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7fd6b5afd67b48f9b0f0597ccd4cf4df', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:08:47.615826 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d7e8975-f5b2-4a3c-8a32-f46eede0af49 service cinder] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:08:47.616031 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d7e8975-f5b2-4a3c-8a32-f46eede0af49 service cinder] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:08:47.616409 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d7e8975-f5b2-4a3c-8a32-f46eede0af49 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:08:47.618929 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-1d7e8975-f5b2-4a3c-8a32-f46eede0af49 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:08:47.685975 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1083/2203] 10.222.0.50 () {58 vars in 958 bytes} [Tue Sep 13 16:08:47 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 123 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:08:47.696607 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d27fe078-c698-425f-b232-52867f4e3fc6 None None] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:08:47.698161 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-d27fe078-c698-425f-b232-52867f4e3fc6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:08:47.710299 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d7e8975-f5b2-4a3c-8a32-f46eede0af49 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:08:47.711916 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d7e8975-f5b2-4a3c-8a32-f46eede0af49 service cinder] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:08:47.724371 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1121/2204] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 16:08:47 2022] GET /identity/v3/auth/tokens => generated 3327 bytes in 261 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 13 16:08:47.770942 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d27fe078-c698-425f-b232-52867f4e3fc6 None None] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:08:47.774723 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a2f821f-9d6b-4d11-a912-2afc43355e60 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f436f07939d4855a21b195bc7566244', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:08:47.775377 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a2f821f-9d6b-4d11-a912-2afc43355e60 service neutron] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:08:47.775659 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a2f821f-9d6b-4d11-a912-2afc43355e60 service neutron] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:08:47.775931 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a2f821f-9d6b-4d11-a912-2afc43355e60 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:08:47.790429 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a2f821f-9d6b-4d11-a912-2afc43355e60 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:08:47.792891 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a2f821f-9d6b-4d11-a912-2afc43355e60 service neutron] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:08:47.808453 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1084/2205] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 16:08:47 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 112 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 13 16:08:49.081982 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-761b24a8-0d0a-4f25-bc38-a82529fbf9d1 None None] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:08:49.082493 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-761b24a8-0d0a-4f25-bc38-a82529fbf9d1 None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:08:49.082573 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-761b24a8-0d0a-4f25-bc38-a82529fbf9d1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:08:49.113067 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.auth.core [None req-761b24a8-0d0a-4f25-bc38-a82529fbf9d1 None None] MFA Rules not processed for user `313ac02befbf44ebabf3b41f84245c1f`. Rule list: `[]` (Enabled: `True`). {{(pid=81437) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:08:49.114308 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-761b24a8-0d0a-4f25-bc38-a82529fbf9d1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:08:49.128223 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:08:49.197934 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1122/2206] 10.222.0.50 () {58 vars in 958 bytes} [Tue Sep 13 16:08:49 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 117 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:08:49.215237 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a2f821f-9d6b-4d11-a912-2afc43355e60 service neutron] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:08:49.225494 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a2f821f-9d6b-4d11-a912-2afc43355e60 service neutron] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:08:49.229895 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-772e5b69-c378-4c04-8b01-31248b7c3f80 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7fd6b5afd67b48f9b0f0597ccd4cf4df', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:08:49.230663 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-772e5b69-c378-4c04-8b01-31248b7c3f80 service cinder] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:08:49.242840 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-772e5b69-c378-4c04-8b01-31248b7c3f80 service cinder] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:08:49.243299 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-772e5b69-c378-4c04-8b01-31248b7c3f80 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:08:49.276246 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-772e5b69-c378-4c04-8b01-31248b7c3f80 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:08:49.278211 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-772e5b69-c378-4c04-8b01-31248b7c3f80 service cinder] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:08:49.295098 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1085/2207] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 16:08:49 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 80 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 16:08:50.812228 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd3e5531-1140-454e-894b-e9e459ff2708 None None] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:08:50.812759 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd3e5531-1140-454e-894b-e9e459ff2708 None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:08:50.812840 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd3e5531-1140-454e-894b-e9e459ff2708 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:08:50.837085 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.auth.core [None req-cd3e5531-1140-454e-894b-e9e459ff2708 None None] MFA Rules not processed for user `313ac02befbf44ebabf3b41f84245c1f`. Rule list: `[]` (Enabled: `True`). {{(pid=81437) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:08:50.839240 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-cd3e5531-1140-454e-894b-e9e459ff2708 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:08:50.848823 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1123/2208] 10.222.0.50 () {58 vars in 958 bytes} [Tue Sep 13 16:08:50 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 38 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:08:52.946156 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-772e5b69-c378-4c04-8b01-31248b7c3f80 service cinder] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:08:52.948596 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-772e5b69-c378-4c04-8b01-31248b7c3f80 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:08:52.985192 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-772e5b69-c378-4c04-8b01-31248b7c3f80 service cinder] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:08:52.991532 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f1a083f-b83a-4bff-a2d7-fd77da568a2d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:08:52.992338 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f1a083f-b83a-4bff-a2d7-fd77da568a2d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:08:52.992612 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f1a083f-b83a-4bff-a2d7-fd77da568a2d admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:08:52.993092 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f1a083f-b83a-4bff-a2d7-fd77da568a2d admin admin] PATH_INFO: `/v3/users/1303fb8c7b774eea88240f942c0c4bee` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:08:52.994802 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f1a083f-b83a-4bff-a2d7-fd77da568a2d admin admin] RBAC: Authorizing `identity:delete_user(user_id=1303fb8c7b774eea88240f942c0c4bee)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:08:52.997572 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f1a083f-b83a-4bff-a2d7-fd77da568a2d admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:08:53.084212 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd3e5531-1140-454e-894b-e9e459ff2708 None None] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:08:53.086227 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-cd3e5531-1140-454e-894b-e9e459ff2708 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:08:53.097575 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-3f1a083f-b83a-4bff-a2d7-fd77da568a2d admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '1303fb8c7b774eea88240f942c0c4bee'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:08:53.106454 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd3e5531-1140-454e-894b-e9e459ff2708 None None] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:08:53.107100 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-3f1a083f-b83a-4bff-a2d7-fd77da568a2d admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '1303fb8c7b774eea88240f942c0c4bee'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:08:53.113030 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42c6cbed-4c84-42ab-a996-a8421fbe59c5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:08:53.113850 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42c6cbed-4c84-42ab-a996-a8421fbe59c5 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:08:53.114237 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42c6cbed-4c84-42ab-a996-a8421fbe59c5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:08:53.114649 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42c6cbed-4c84-42ab-a996-a8421fbe59c5 admin admin] PATH_INFO: `/v3/users/9e14c4b1e41544f1862cd85c60091bc7` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:08:53.115326 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=89,keystone:INSERT=2,keystone:DELETE=9 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:08:53.116234 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42c6cbed-4c84-42ab-a996-a8421fbe59c5 admin admin] RBAC: Authorizing `identity:delete_user(user_id=9e14c4b1e41544f1862cd85c60091bc7)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:08:53.118584 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42c6cbed-4c84-42ab-a996-a8421fbe59c5 admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:08:53.126157 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-3f1a083f-b83a-4bff-a2d7-fd77da568a2d admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '1303fb8c7b774eea88240f942c0c4bee'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:08:53.127013 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-3f1a083f-b83a-4bff-a2d7-fd77da568a2d admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '1303fb8c7b774eea88240f942c0c4bee'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:08:53.135815 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-3f1a083f-b83a-4bff-a2d7-fd77da568a2d admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '1303fb8c7b774eea88240f942c0c4bee'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:08:53.141947 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1086/2209] 10.222.0.50 () {58 vars in 1224 bytes} [Tue Sep 13 16:08:52 2022] DELETE /identity/v3/users/1303fb8c7b774eea88240f942c0c4bee => generated 0 bytes in 196 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 16:08:53.147149 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ceacf2a2-2437-474e-8b73-ce0f0e883bf4 None None] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:08:53.147437 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ceacf2a2-2437-474e-8b73-ce0f0e883bf4 None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:08:53.147766 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ceacf2a2-2437-474e-8b73-ce0f0e883bf4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:08:53.174482 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.auth.core [None req-ceacf2a2-2437-474e-8b73-ce0f0e883bf4 None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81438) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:08:53.175997 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-ceacf2a2-2437-474e-8b73-ce0f0e883bf4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:08:53.178320 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-42c6cbed-4c84-42ab-a996-a8421fbe59c5 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '9e14c4b1e41544f1862cd85c60091bc7'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:08:53.187163 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-42c6cbed-4c84-42ab-a996-a8421fbe59c5 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '9e14c4b1e41544f1862cd85c60091bc7'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:08:53.204237 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-42c6cbed-4c84-42ab-a996-a8421fbe59c5 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '9e14c4b1e41544f1862cd85c60091bc7'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:08:53.205750 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-42c6cbed-4c84-42ab-a996-a8421fbe59c5 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '9e14c4b1e41544f1862cd85c60091bc7'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:08:53.217777 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-42c6cbed-4c84-42ab-a996-a8421fbe59c5 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '9e14c4b1e41544f1862cd85c60091bc7'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:08:53.228755 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1124/2210] 10.222.0.50 () {58 vars in 1224 bytes} [Tue Sep 13 16:08:53 2022] DELETE /identity/v3/users/9e14c4b1e41544f1862cd85c60091bc7 => generated 0 bytes in 146 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 13 16:08:53.249248 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1087/2211] 10.222.0.50 () {58 vars in 958 bytes} [Tue Sep 13 16:08:53 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 13 16:08:53.262763 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42c6cbed-4c84-42ab-a996-a8421fbe59c5 admin admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:08:53.264735 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-42c6cbed-4c84-42ab-a996-a8421fbe59c5 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:08:53.346716 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42c6cbed-4c84-42ab-a996-a8421fbe59c5 admin admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:08:53.353801 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5977a3da-b014-4240-90b7-aaa177ed2e95 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f436f07939d4855a21b195bc7566244', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:08:53.354979 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5977a3da-b014-4240-90b7-aaa177ed2e95 service neutron] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:08:53.355557 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5977a3da-b014-4240-90b7-aaa177ed2e95 service neutron] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:08:53.356708 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5977a3da-b014-4240-90b7-aaa177ed2e95 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:08:53.403971 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:08:53.448991 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ceacf2a2-2437-474e-8b73-ce0f0e883bf4 None None] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:08:53.451089 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-ceacf2a2-2437-474e-8b73-ce0f0e883bf4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:08:53.458961 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5977a3da-b014-4240-90b7-aaa177ed2e95 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:08:53.461465 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5977a3da-b014-4240-90b7-aaa177ed2e95 service neutron] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:08:53.476894 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1125/2212] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 16:08:53 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 215 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 13 16:08:53.483538 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ceacf2a2-2437-474e-8b73-ce0f0e883bf4 None None] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:08:53.490736 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f542d9e-f2ca-449a-82c8-3c6660a5d314 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:08:53.491558 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f542d9e-f2ca-449a-82c8-3c6660a5d314 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:08:53.491850 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f542d9e-f2ca-449a-82c8-3c6660a5d314 admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:08:53.492126 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f542d9e-f2ca-449a-82c8-3c6660a5d314 admin admin] PATH_INFO: `/v3/projects/04650bcf4da349c184b993cbaadb6871` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:08:53.493332 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f542d9e-f2ca-449a-82c8-3c6660a5d314 admin admin] RBAC: Authorizing `identity:delete_project(project_id=04650bcf4da349c184b993cbaadb6871)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:08:53.495318 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f542d9e-f2ca-449a-82c8-3c6660a5d314 admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:08:53.556911 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-1f542d9e-f2ca-449a-82c8-3c6660a5d314 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '04650bcf4da349c184b993cbaadb6871', 'request_id': 'req-1f542d9e-f2ca-449a-82c8-3c6660a5d314'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:08:53.567321 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-1f542d9e-f2ca-449a-82c8-3c6660a5d314 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '04650bcf4da349c184b993cbaadb6871', 'request_id': 'req-1f542d9e-f2ca-449a-82c8-3c6660a5d314'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:08:53.575597 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-1f542d9e-f2ca-449a-82c8-3c6660a5d314 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '04650bcf4da349c184b993cbaadb6871', 'request_id': 'req-1f542d9e-f2ca-449a-82c8-3c6660a5d314'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:08:53.592898 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-1f542d9e-f2ca-449a-82c8-3c6660a5d314 admin admin] The token cache is being invalidate because project 04650bcf4da349c184b993cbaadb6871 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81438) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 13 16:08:53.593242 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-1f542d9e-f2ca-449a-82c8-3c6660a5d314 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:08:53.593986 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-1f542d9e-f2ca-449a-82c8-3c6660a5d314 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:08:53.595831 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1088/2213] 10.222.0.50 () {58 vars in 1233 bytes} [Tue Sep 13 16:08:53 2022] DELETE /identity/v3/projects/04650bcf4da349c184b993cbaadb6871 => generated 0 bytes in 147 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 13 16:08:53.599937 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5977a3da-b014-4240-90b7-aaa177ed2e95 service neutron] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:08:53.610431 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5977a3da-b014-4240-90b7-aaa177ed2e95 service neutron] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:08:53.728731 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6724da58-aea5-4c83-a97d-7ac6b1489727 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:08:53.732084 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6724da58-aea5-4c83-a97d-7ac6b1489727 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:08:53.733092 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6724da58-aea5-4c83-a97d-7ac6b1489727 admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:08:53.733718 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6724da58-aea5-4c83-a97d-7ac6b1489727 admin admin] PATH_INFO: `/v3/users/05db30cc72f74957b378a05672507630` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:08:53.736573 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6724da58-aea5-4c83-a97d-7ac6b1489727 admin admin] RBAC: Authorizing `identity:delete_user(user_id=05db30cc72f74957b378a05672507630)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:08:53.745477 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6724da58-aea5-4c83-a97d-7ac6b1489727 admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:08:53.810395 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-6724da58-aea5-4c83-a97d-7ac6b1489727 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '05db30cc72f74957b378a05672507630'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:08:53.822987 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-6724da58-aea5-4c83-a97d-7ac6b1489727 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '05db30cc72f74957b378a05672507630'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:08:53.841032 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-6724da58-aea5-4c83-a97d-7ac6b1489727 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '05db30cc72f74957b378a05672507630'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:08:53.841793 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-6724da58-aea5-4c83-a97d-7ac6b1489727 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '05db30cc72f74957b378a05672507630'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:08:53.851812 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-6724da58-aea5-4c83-a97d-7ac6b1489727 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '05db30cc72f74957b378a05672507630'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:08:53.866985 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1126/2214] 10.222.0.50 () {58 vars in 1224 bytes} [Tue Sep 13 16:08:53 2022] DELETE /identity/v3/users/05db30cc72f74957b378a05672507630 => generated 0 bytes in 265 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 16:08:54.051346 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f542d9e-f2ca-449a-82c8-3c6660a5d314 admin admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:08:54.054511 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-1f542d9e-f2ca-449a-82c8-3c6660a5d314 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:08:54.076886 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f542d9e-f2ca-449a-82c8-3c6660a5d314 admin admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:08:54.155247 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0667b31-fa1d-40f3-ad6d-212cbd55b296 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:08:54.156824 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0667b31-fa1d-40f3-ad6d-212cbd55b296 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:08:54.156824 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0667b31-fa1d-40f3-ad6d-212cbd55b296 admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:08:54.156824 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0667b31-fa1d-40f3-ad6d-212cbd55b296 admin admin] PATH_INFO: `/v3/projects/b30c55812b704372a65bfd70422d0cff` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:08:54.157833 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0667b31-fa1d-40f3-ad6d-212cbd55b296 admin admin] RBAC: Authorizing `identity:delete_project(project_id=b30c55812b704372a65bfd70422d0cff)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:08:54.160475 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0667b31-fa1d-40f3-ad6d-212cbd55b296 admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:08:54.168406 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=91,keystone:DELETE=8,keystone:INSERT=1 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:08:54.224615 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-f0667b31-fa1d-40f3-ad6d-212cbd55b296 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'b30c55812b704372a65bfd70422d0cff', 'request_id': 'req-f0667b31-fa1d-40f3-ad6d-212cbd55b296'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:08:54.236738 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-f0667b31-fa1d-40f3-ad6d-212cbd55b296 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'b30c55812b704372a65bfd70422d0cff', 'request_id': 'req-f0667b31-fa1d-40f3-ad6d-212cbd55b296'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:08:54.245877 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-f0667b31-fa1d-40f3-ad6d-212cbd55b296 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'b30c55812b704372a65bfd70422d0cff', 'request_id': 'req-f0667b31-fa1d-40f3-ad6d-212cbd55b296'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:08:54.268542 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-f0667b31-fa1d-40f3-ad6d-212cbd55b296 admin admin] The token cache is being invalidate because project b30c55812b704372a65bfd70422d0cff was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81438) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 13 16:08:54.268867 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-f0667b31-fa1d-40f3-ad6d-212cbd55b296 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:08:54.269500 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-f0667b31-fa1d-40f3-ad6d-212cbd55b296 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:08:54.271295 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1089/2215] 10.222.0.50 () {58 vars in 1233 bytes} [Tue Sep 13 16:08:54 2022] DELETE /identity/v3/projects/b30c55812b704372a65bfd70422d0cff => generated 0 bytes in 223 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 16:08:54.275540 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6724da58-aea5-4c83-a97d-7ac6b1489727 admin admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:08:54.277014 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-6724da58-aea5-4c83-a97d-7ac6b1489727 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:08:54.290581 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6724da58-aea5-4c83-a97d-7ac6b1489727 admin admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:08:54.349995 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:08:54.363509 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fa8d888-2f45-4065-b173-6d1262792b3c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:08:54.364107 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fa8d888-2f45-4065-b173-6d1262792b3c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:08:54.364716 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fa8d888-2f45-4065-b173-6d1262792b3c admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:08:54.365144 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fa8d888-2f45-4065-b173-6d1262792b3c admin admin] PATH_INFO: `/v3/users/a82bb8e5858a4f49a95a8f01bdf76019` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:08:54.366602 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fa8d888-2f45-4065-b173-6d1262792b3c admin admin] RBAC: Authorizing `identity:delete_user(user_id=a82bb8e5858a4f49a95a8f01bdf76019)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:08:54.368677 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fa8d888-2f45-4065-b173-6d1262792b3c admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:08:54.418752 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-0fa8d888-2f45-4065-b173-6d1262792b3c admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'a82bb8e5858a4f49a95a8f01bdf76019'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:08:54.426615 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-0fa8d888-2f45-4065-b173-6d1262792b3c admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'a82bb8e5858a4f49a95a8f01bdf76019'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:08:54.440999 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-0fa8d888-2f45-4065-b173-6d1262792b3c admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'a82bb8e5858a4f49a95a8f01bdf76019'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:08:54.441738 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-0fa8d888-2f45-4065-b173-6d1262792b3c admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'a82bb8e5858a4f49a95a8f01bdf76019'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:08:54.449576 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-0fa8d888-2f45-4065-b173-6d1262792b3c admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'a82bb8e5858a4f49a95a8f01bdf76019'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:08:54.457782 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1127/2216] 10.222.0.50 () {58 vars in 1224 bytes} [Tue Sep 13 16:08:54 2022] DELETE /identity/v3/users/a82bb8e5858a4f49a95a8f01bdf76019 => generated 0 bytes in 183 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 16:08:54.539251 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0667b31-fa1d-40f3-ad6d-212cbd55b296 admin admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:08:54.540796 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-f0667b31-fa1d-40f3-ad6d-212cbd55b296 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:08:54.554949 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0667b31-fa1d-40f3-ad6d-212cbd55b296 admin admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:08:54.641842 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5301597d-c5ff-4863-b660-7c2e78a75268 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:08:54.642843 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5301597d-c5ff-4863-b660-7c2e78a75268 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:08:54.643298 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5301597d-c5ff-4863-b660-7c2e78a75268 admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:08:54.643731 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5301597d-c5ff-4863-b660-7c2e78a75268 admin admin] PATH_INFO: `/v3/projects/d3262fc47ea44842a292beb228bd256d` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:08:54.645016 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5301597d-c5ff-4863-b660-7c2e78a75268 admin admin] RBAC: Authorizing `identity:delete_project(project_id=d3262fc47ea44842a292beb228bd256d)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:08:54.649582 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5301597d-c5ff-4863-b660-7c2e78a75268 admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:08:54.718029 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-5301597d-c5ff-4863-b660-7c2e78a75268 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'd3262fc47ea44842a292beb228bd256d', 'request_id': 'req-5301597d-c5ff-4863-b660-7c2e78a75268'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:08:54.730755 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-5301597d-c5ff-4863-b660-7c2e78a75268 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'd3262fc47ea44842a292beb228bd256d', 'request_id': 'req-5301597d-c5ff-4863-b660-7c2e78a75268'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:08:54.741091 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-5301597d-c5ff-4863-b660-7c2e78a75268 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'd3262fc47ea44842a292beb228bd256d', 'request_id': 'req-5301597d-c5ff-4863-b660-7c2e78a75268'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:08:54.756584 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-5301597d-c5ff-4863-b660-7c2e78a75268 admin admin] The token cache is being invalidate because project d3262fc47ea44842a292beb228bd256d was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81438) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 13 16:08:54.757150 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-5301597d-c5ff-4863-b660-7c2e78a75268 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:08:54.758019 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-5301597d-c5ff-4863-b660-7c2e78a75268 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:08:54.759928 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1090/2217] 10.222.0.50 () {58 vars in 1233 bytes} [Tue Sep 13 16:08:54 2022] DELETE /identity/v3/projects/d3262fc47ea44842a292beb228bd256d => generated 0 bytes in 221 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 16:08:54.765008 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e14d2916-6075-40d0-ac94-3f38101757ad None None] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:08:54.765286 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e14d2916-6075-40d0-ac94-3f38101757ad None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:08:54.765559 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e14d2916-6075-40d0-ac94-3f38101757ad None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:08:54.788160 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.auth.core [None req-e14d2916-6075-40d0-ac94-3f38101757ad None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81437) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:08:54.789492 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-e14d2916-6075-40d0-ac94-3f38101757ad None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:08:54.812161 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5301597d-c5ff-4863-b660-7c2e78a75268 admin admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:08:54.813880 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-5301597d-c5ff-4863-b660-7c2e78a75268 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:08:54.867511 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=84,keystone:DELETE=14,keystone:INSERT=2 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:08:54.916773 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1128/2218] 10.222.0.50 () {58 vars in 958 bytes} [Tue Sep 13 16:08:54 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 153 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:08:54.918579 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce84a45e-7c34-4204-a2e0-061b6c3d1a78 None None] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:08:54.918870 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce84a45e-7c34-4204-a2e0-061b6c3d1a78 None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:08:54.919310 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce84a45e-7c34-4204-a2e0-061b6c3d1a78 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:08:54.951369 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5301597d-c5ff-4863-b660-7c2e78a75268 admin admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:08:54.958916 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eff2e982-928c-41f4-8a77-1ec74d180e8d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:08:54.959584 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eff2e982-928c-41f4-8a77-1ec74d180e8d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:08:54.959864 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eff2e982-928c-41f4-8a77-1ec74d180e8d admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:08:54.960142 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eff2e982-928c-41f4-8a77-1ec74d180e8d admin admin] PATH_INFO: `/v3/projects/a03553ba854c438fbd0d02b5c608d975` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:08:54.961316 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eff2e982-928c-41f4-8a77-1ec74d180e8d admin admin] RBAC: Authorizing `identity:delete_project(project_id=a03553ba854c438fbd0d02b5c608d975)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:08:54.963424 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eff2e982-928c-41f4-8a77-1ec74d180e8d admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:08:54.972576 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.auth.core [None req-ce84a45e-7c34-4204-a2e0-061b6c3d1a78 None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81437) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:08:54.973808 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-ce84a45e-7c34-4204-a2e0-061b6c3d1a78 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:08:54.991776 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1129/2219] 10.222.0.50 () {58 vars in 958 bytes} [Tue Sep 13 16:08:54 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 74 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:08:54.996080 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce84a45e-7c34-4204-a2e0-061b6c3d1a78 None None] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:08:55.013163 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce84a45e-7c34-4204-a2e0-061b6c3d1a78 None None] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:08:55.035935 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-eff2e982-928c-41f4-8a77-1ec74d180e8d admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'a03553ba854c438fbd0d02b5c608d975', 'request_id': 'req-eff2e982-928c-41f4-8a77-1ec74d180e8d'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:08:55.045661 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-eff2e982-928c-41f4-8a77-1ec74d180e8d admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'a03553ba854c438fbd0d02b5c608d975', 'request_id': 'req-eff2e982-928c-41f4-8a77-1ec74d180e8d'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:08:55.060104 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-eff2e982-928c-41f4-8a77-1ec74d180e8d admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'a03553ba854c438fbd0d02b5c608d975', 'request_id': 'req-eff2e982-928c-41f4-8a77-1ec74d180e8d'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:08:55.075799 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-eff2e982-928c-41f4-8a77-1ec74d180e8d admin admin] The token cache is being invalidate because project a03553ba854c438fbd0d02b5c608d975 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81438) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 13 16:08:55.076096 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-eff2e982-928c-41f4-8a77-1ec74d180e8d admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:08:55.076965 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-eff2e982-928c-41f4-8a77-1ec74d180e8d admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:08:55.083326 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1091/2220] 10.222.0.50 () {58 vars in 1233 bytes} [Tue Sep 13 16:08:54 2022] DELETE /identity/v3/projects/a03553ba854c438fbd0d02b5c608d975 => generated 0 bytes in 267 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 13 16:08:55.085533 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f93cd876-2926-4364-a0a8-c66ca0578932 None None] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:08:55.086010 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f93cd876-2926-4364-a0a8-c66ca0578932 None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:08:55.086278 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f93cd876-2926-4364-a0a8-c66ca0578932 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:08:55.109616 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.auth.core [None req-f93cd876-2926-4364-a0a8-c66ca0578932 None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81438) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:08:55.110859 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-f93cd876-2926-4364-a0a8-c66ca0578932 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:08:55.113565 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bff3dae-b900-4ce3-a600-caa7d9e33b6c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:08:55.114182 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bff3dae-b900-4ce3-a600-caa7d9e33b6c admin admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:08:55.114460 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bff3dae-b900-4ce3-a600-caa7d9e33b6c admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:08:55.114728 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bff3dae-b900-4ce3-a600-caa7d9e33b6c admin admin] PATH_INFO: `/v3/domains` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:08:55.115439 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1bff3dae-b900-4ce3-a600-caa7d9e33b6c admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81437) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 16:08:55.115766 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1bff3dae-b900-4ce3-a600-caa7d9e33b6c admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:08:55.117364 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1bff3dae-b900-4ce3-a600-caa7d9e33b6c admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:08:55.125475 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:08:55.128421 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1092/2221] 10.222.0.50 () {58 vars in 958 bytes} [Tue Sep 13 16:08:55 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 44 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:08:55.130853 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b85a5983-c7be-479b-aa81-67764c1d0782 None None] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:08:55.131241 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b85a5983-c7be-479b-aa81-67764c1d0782 None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:08:55.131597 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b85a5983-c7be-479b-aa81-67764c1d0782 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:08:55.136277 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1130/2222] 10.222.0.50 () {58 vars in 1153 bytes} [Tue Sep 13 16:08:54 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 13 16:08:55.140787 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bff3dae-b900-4ce3-a600-caa7d9e33b6c admin admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:08:55.146982 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-1bff3dae-b900-4ce3-a600-caa7d9e33b6c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:08:55.158273 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.auth.core [None req-b85a5983-c7be-479b-aa81-67764c1d0782 None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81438) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:08:55.160428 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-b85a5983-c7be-479b-aa81-67764c1d0782 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:08:55.175698 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bff3dae-b900-4ce3-a600-caa7d9e33b6c admin admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:08:55.179485 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1093/2223] 10.222.0.50 () {58 vars in 958 bytes} [Tue Sep 13 16:08:55 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 51 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:08:55.183992 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b85a5983-c7be-479b-aa81-67764c1d0782 None None] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:08:55.184272 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a0a59a8-0d7e-4097-bd4e-9b053f48d8df admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:08:55.184894 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a0a59a8-0d7e-4097-bd4e-9b053f48d8df admin admin] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:08:55.185154 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a0a59a8-0d7e-4097-bd4e-9b053f48d8df admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:08:55.185421 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a0a59a8-0d7e-4097-bd4e-9b053f48d8df admin admin] PATH_INFO: `/v3/projects` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:08:55.186050 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a0a59a8-0d7e-4097-bd4e-9b053f48d8df admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:08:55.187996 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a0a59a8-0d7e-4097-bd4e-9b053f48d8df admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:08:55.199803 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b85a5983-c7be-479b-aa81-67764c1d0782 None None] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:08:55.205235 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d94d99cc-3923-4526-bccd-7d4a07091595 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:08:55.205976 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d94d99cc-3923-4526-bccd-7d4a07091595 admin admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:08:55.206330 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d94d99cc-3923-4526-bccd-7d4a07091595 admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:08:55.206672 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d94d99cc-3923-4526-bccd-7d4a07091595 admin admin] PATH_INFO: `/v3/domains` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:08:55.207506 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d94d99cc-3923-4526-bccd-7d4a07091595 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81438) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 16:08:55.207888 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d94d99cc-3923-4526-bccd-7d4a07091595 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:08:55.209553 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d94d99cc-3923-4526-bccd-7d4a07091595 admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:08:55.222891 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1094/2224] 10.222.0.50 () {58 vars in 1153 bytes} [Tue Sep 13 16:08:55 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 16:08:55.225729 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1131/2225] 10.222.0.50 () {60 vars in 1153 bytes} [Tue Sep 13 16:08:55 2022] POST /identity/v3/projects => generated 377 bytes in 85 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 16:08:55.227170 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d94d99cc-3923-4526-bccd-7d4a07091595 admin admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:08:55.229855 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a0a59a8-0d7e-4097-bd4e-9b053f48d8df admin admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:08:55.239807 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d94d99cc-3923-4526-bccd-7d4a07091595 admin admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:08:55.316065 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02f6155d-2030-4660-a05a-c7ac0277a854 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:08:55.316709 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a0a59a8-0d7e-4097-bd4e-9b053f48d8df admin admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:08:55.318335 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02f6155d-2030-4660-a05a-c7ac0277a854 admin admin] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:08:55.319082 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02f6155d-2030-4660-a05a-c7ac0277a854 admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:08:55.319998 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02f6155d-2030-4660-a05a-c7ac0277a854 admin admin] PATH_INFO: `/v3/projects` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:08:55.320994 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02f6155d-2030-4660-a05a-c7ac0277a854 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:08:55.323931 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02f6155d-2030-4660-a05a-c7ac0277a854 admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:08:55.337327 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c16ef029-187d-47bb-a5b1-60768fb928a0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:08:55.338092 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c16ef029-187d-47bb-a5b1-60768fb928a0 admin admin] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:08:55.340523 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c16ef029-187d-47bb-a5b1-60768fb928a0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:08:55.340923 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c16ef029-187d-47bb-a5b1-60768fb928a0 admin admin] PATH_INFO: `/v3/users` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:08:55.341677 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c16ef029-187d-47bb-a5b1-60768fb928a0 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:08:55.347174 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c16ef029-187d-47bb-a5b1-60768fb928a0 admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:08:55.370705 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1095/2226] 10.222.0.50 () {60 vars in 1153 bytes} [Tue Sep 13 16:08:55 2022] POST /identity/v3/projects => generated 363 bytes in 141 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 13 16:08:55.371918 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02f6155d-2030-4660-a05a-c7ac0277a854 admin admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:08:55.389138 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1132/2227] 10.222.0.50 () {60 vars in 1144 bytes} [Tue Sep 13 16:08:55 2022] POST /identity/v3/users => generated 289 bytes in 159 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 13 16:08:55.390243 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c16ef029-187d-47bb-a5b1-60768fb928a0 admin admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:08:55.398540 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:08:55.408544 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c16ef029-187d-47bb-a5b1-60768fb928a0 admin admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:08:55.489237 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-797f3d4e-40c4-46e3-98ff-7849bbadc545 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:08:55.489858 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-797f3d4e-40c4-46e3-98ff-7849bbadc545 admin admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:08:55.490143 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-797f3d4e-40c4-46e3-98ff-7849bbadc545 admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:08:55.490420 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-797f3d4e-40c4-46e3-98ff-7849bbadc545 admin admin] PATH_INFO: `/v3/roles` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:08:55.491141 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-797f3d4e-40c4-46e3-98ff-7849bbadc545 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:08:55.493048 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-797f3d4e-40c4-46e3-98ff-7849bbadc545 admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:08:55.504906 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1133/2228] 10.222.0.50 () {58 vars in 1122 bytes} [Tue Sep 13 16:08:55 2022] GET /identity/v3/roles => generated 1420 bytes in 115 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 16:08:55.506629 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02f6155d-2030-4660-a05a-c7ac0277a854 admin admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:08:55.508701 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-797f3d4e-40c4-46e3-98ff-7849bbadc545 admin admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:08:55.513773 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb6ad9e3-2746-4543-8245-cd404ec3183e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:08:55.514533 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb6ad9e3-2746-4543-8245-cd404ec3183e admin admin] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:08:55.515894 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb6ad9e3-2746-4543-8245-cd404ec3183e admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:08:55.516186 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb6ad9e3-2746-4543-8245-cd404ec3183e admin admin] PATH_INFO: `/v3/users` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:08:55.516829 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb6ad9e3-2746-4543-8245-cd404ec3183e admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:08:55.519082 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-797f3d4e-40c4-46e3-98ff-7849bbadc545 admin admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:08:55.519474 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb6ad9e3-2746-4543-8245-cd404ec3183e admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:08:55.524178 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54759f39-39a1-43df-9498-f8ffc705969f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:08:55.524858 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54759f39-39a1-43df-9498-f8ffc705969f admin admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:08:55.525215 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54759f39-39a1-43df-9498-f8ffc705969f admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:08:55.525564 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54759f39-39a1-43df-9498-f8ffc705969f admin admin] PATH_INFO: `/v3/roles` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:08:55.526283 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54759f39-39a1-43df-9498-f8ffc705969f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:08:55.528006 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54759f39-39a1-43df-9498-f8ffc705969f admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:08:55.539895 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1134/2229] 10.222.0.50 () {58 vars in 1122 bytes} [Tue Sep 13 16:08:55 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 13 16:08:55.543920 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54759f39-39a1-43df-9498-f8ffc705969f admin admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:08:55.552029 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1096/2230] 10.222.0.50 () {60 vars in 1144 bytes} [Tue Sep 13 16:08:55 2022] POST /identity/v3/users => generated 282 bytes in 176 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 16:08:55.553207 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb6ad9e3-2746-4543-8245-cd404ec3183e admin admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:08:55.558080 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54759f39-39a1-43df-9498-f8ffc705969f admin admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:08:55.563199 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6864bd9-aed6-4cea-9893-f2e7ae408e92 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:08:55.563916 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6864bd9-aed6-4cea-9893-f2e7ae408e92 admin admin] REQUEST_METHOD: `PUT` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:08:55.564770 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6864bd9-aed6-4cea-9893-f2e7ae408e92 admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:08:55.565338 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6864bd9-aed6-4cea-9893-f2e7ae408e92 admin admin] PATH_INFO: `/v3/projects/991991dd8a1d4c4bb8c6fc4ea68ad9b8/users/155450dfe7034865821a0472f1ba0966/roles/15db4aaca3264ae29bde38903ba43471` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:08:55.567498 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb6ad9e3-2746-4543-8245-cd404ec3183e admin admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:08:55.574482 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc6fd62e-076d-4455-97e3-7306e64ded22 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:08:55.575008 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc6fd62e-076d-4455-97e3-7306e64ded22 admin admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:08:55.575293 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc6fd62e-076d-4455-97e3-7306e64ded22 admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:08:55.575710 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc6fd62e-076d-4455-97e3-7306e64ded22 admin admin] PATH_INFO: `/v3/roles` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:08:55.576395 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc6fd62e-076d-4455-97e3-7306e64ded22 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:08:55.578399 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc6fd62e-076d-4455-97e3-7306e64ded22 admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:08:55.588388 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6864bd9-aed6-4cea-9893-f2e7ae408e92 admin admin] RBAC: Authorizing `identity:create_grant(project_id=991991dd8a1d4c4bb8c6fc4ea68ad9b8, user_id=155450dfe7034865821a0472f1ba0966, role_id=15db4aaca3264ae29bde38903ba43471)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:08:55.591160 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1097/2231] 10.222.0.50 () {58 vars in 1122 bytes} [Tue Sep 13 16:08:55 2022] GET /identity/v3/roles => generated 1420 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 16:08:55.591744 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6864bd9-aed6-4cea-9893-f2e7ae408e92 admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:08:55.596687 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc6fd62e-076d-4455-97e3-7306e64ded22 admin admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:08:55.608438 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc6fd62e-076d-4455-97e3-7306e64ded22 admin admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:08:55.614948 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1135/2232] 10.222.0.50 () {60 vars in 1483 bytes} [Tue Sep 13 16:08:55 2022] PUT /identity/v3/projects/991991dd8a1d4c4bb8c6fc4ea68ad9b8/users/155450dfe7034865821a0472f1ba0966/roles/15db4aaca3264ae29bde38903ba43471 => generated 0 bytes in 71 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 13 16:08:55.626632 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51a44e30-c50c-4461-8b96-d45d495e8872 None None] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:08:55.627508 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51a44e30-c50c-4461-8b96-d45d495e8872 None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:08:55.627508 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51a44e30-c50c-4461-8b96-d45d495e8872 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:08:55.658532 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.auth.core [None req-51a44e30-c50c-4461-8b96-d45d495e8872 None None] MFA Rules not processed for user `155450dfe7034865821a0472f1ba0966`. Rule list: `[]` (Enabled: `True`). {{(pid=81437) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:08:55.659895 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-51a44e30-c50c-4461-8b96-d45d495e8872 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:08:55.673344 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:08:55.678251 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c17f495b-4913-479f-98c7-0f60ea682739 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:08:55.678888 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c17f495b-4913-479f-98c7-0f60ea682739 admin admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:08:55.679193 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c17f495b-4913-479f-98c7-0f60ea682739 admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:08:55.679465 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c17f495b-4913-479f-98c7-0f60ea682739 admin admin] PATH_INFO: `/v3/roles` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:08:55.680102 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c17f495b-4913-479f-98c7-0f60ea682739 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:08:55.681844 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c17f495b-4913-479f-98c7-0f60ea682739 admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:08:55.692010 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1098/2233] 10.222.0.50 () {58 vars in 1122 bytes} [Tue Sep 13 16:08:55 2022] GET /identity/v3/roles => generated 1420 bytes in 97 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 16:08:55.696017 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c17f495b-4913-479f-98c7-0f60ea682739 admin admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:08:55.705855 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c17f495b-4913-479f-98c7-0f60ea682739 admin admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:08:55.710712 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8a67bc6-0e9d-4593-86a6-7841d8962643 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:08:55.711440 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8a67bc6-0e9d-4593-86a6-7841d8962643 admin admin] REQUEST_METHOD: `PUT` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:08:55.711725 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8a67bc6-0e9d-4593-86a6-7841d8962643 admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:08:55.711999 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8a67bc6-0e9d-4593-86a6-7841d8962643 admin admin] PATH_INFO: `/v3/projects/8c9e03206f954da5a29ab5d6e9881745/users/e06b490d80174c7a974c94b4c8b38a99/roles/15db4aaca3264ae29bde38903ba43471` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:08:55.727679 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8a67bc6-0e9d-4593-86a6-7841d8962643 admin admin] RBAC: Authorizing `identity:create_grant(project_id=8c9e03206f954da5a29ab5d6e9881745, user_id=e06b490d80174c7a974c94b4c8b38a99, role_id=15db4aaca3264ae29bde38903ba43471)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:08:55.730341 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8a67bc6-0e9d-4593-86a6-7841d8962643 admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:08:55.730469 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1136/2234] 10.222.0.50 () {58 vars in 958 bytes} [Tue Sep 13 16:08:55 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:08:55.743314 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51a44e30-c50c-4461-8b96-d45d495e8872 None None] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:08:55.744762 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-51a44e30-c50c-4461-8b96-d45d495e8872 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:08:55.748129 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1099/2235] 10.222.0.50 () {60 vars in 1483 bytes} [Tue Sep 13 16:08:55 2022] PUT /identity/v3/projects/8c9e03206f954da5a29ab5d6e9881745/users/e06b490d80174c7a974c94b4c8b38a99/roles/15db4aaca3264ae29bde38903ba43471 => generated 0 bytes in 52 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 13 16:08:55.761389 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74eb84df-6002-402f-875e-786f6794cd75 None None] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:08:55.761676 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74eb84df-6002-402f-875e-786f6794cd75 None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:08:55.761951 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74eb84df-6002-402f-875e-786f6794cd75 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:08:55.792350 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.auth.core [None req-74eb84df-6002-402f-875e-786f6794cd75 None None] MFA Rules not processed for user `e06b490d80174c7a974c94b4c8b38a99`. Rule list: `[]` (Enabled: `True`). {{(pid=81438) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:08:55.793838 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-74eb84df-6002-402f-875e-786f6794cd75 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:08:55.809704 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51a44e30-c50c-4461-8b96-d45d495e8872 None None] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:08:55.843570 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:08:55.859069 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be23d57f-c0e2-4c6b-8b32-f0142af8a1c2 service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6ab5f8018f464be69d4af7683f553738', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:08:55.859668 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be23d57f-c0e2-4c6b-8b32-f0142af8a1c2 service glance] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:08:55.859944 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be23d57f-c0e2-4c6b-8b32-f0142af8a1c2 service glance] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:08:55.860212 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be23d57f-c0e2-4c6b-8b32-f0142af8a1c2 service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:08:55.862148 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1100/2236] 10.222.0.50 () {58 vars in 958 bytes} [Tue Sep 13 16:08:55 2022] POST /identity/v3/auth/tokens => generated 3240 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 13 16:08:55.873512 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74eb84df-6002-402f-875e-786f6794cd75 None None] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:08:55.874936 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-74eb84df-6002-402f-875e-786f6794cd75 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:08:55.918739 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be23d57f-c0e2-4c6b-8b32-f0142af8a1c2 service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:08:55.920212 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be23d57f-c0e2-4c6b-8b32-f0142af8a1c2 service glance] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:08:55.929675 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1137/2237] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 16:08:55 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 187 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 13 16:08:55.932275 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74eb84df-6002-402f-875e-786f6794cd75 None None] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:08:55.935906 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7f7f6fa-877c-4f3d-8c78-3765ae6e194e service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7fd6b5afd67b48f9b0f0597ccd4cf4df', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:08:55.936514 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7f7f6fa-877c-4f3d-8c78-3765ae6e194e service cinder] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:08:55.936797 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7f7f6fa-877c-4f3d-8c78-3765ae6e194e service cinder] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:08:55.937070 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7f7f6fa-877c-4f3d-8c78-3765ae6e194e service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:08:55.945924 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7f7f6fa-877c-4f3d-8c78-3765ae6e194e service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:08:55.947505 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7f7f6fa-877c-4f3d-8c78-3765ae6e194e service cinder] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:08:55.956681 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1101/2238] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 16:08:55 2022] GET /identity/v3/auth/tokens => generated 3240 bytes in 84 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 13 16:09:05.925638 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=88 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:09:05.952471 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=30 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:09:14.902553 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30ddd18a-d6b0-40f6-8242-c9a5d54eeac3 None None] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:09:14.903088 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30ddd18a-d6b0-40f6-8242-c9a5d54eeac3 None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:09:14.903088 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30ddd18a-d6b0-40f6-8242-c9a5d54eeac3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:09:14.923314 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.auth.core [None req-30ddd18a-d6b0-40f6-8242-c9a5d54eeac3 None None] MFA Rules not processed for user `6d31b2ec391441f7bab3546d2354b0a1`. Rule list: `[]` (Enabled: `True`). {{(pid=81437) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:09:14.924522 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-30ddd18a-d6b0-40f6-8242-c9a5d54eeac3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:09:14.979953 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1138/2239] 10.222.0.50 () {58 vars in 958 bytes} [Tue Sep 13 16:09:14 2022] POST /identity/v3/auth/tokens => generated 3268 bytes in 79 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:09:24.972892 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=25 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:09:28.735969 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7f7f6fa-877c-4f3d-8c78-3765ae6e194e service cinder] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:09:28.804170 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7f7f6fa-877c-4f3d-8c78-3765ae6e194e service cinder] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:09:28.809058 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef79984e-5611-45c4-9b80-3859fd363698 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:09:28.809688 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef79984e-5611-45c4-9b80-3859fd363698 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:09:28.809968 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef79984e-5611-45c4-9b80-3859fd363698 admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:09:28.810244 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef79984e-5611-45c4-9b80-3859fd363698 admin admin] PATH_INFO: `/v3/users/155450dfe7034865821a0472f1ba0966` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:09:28.811400 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef79984e-5611-45c4-9b80-3859fd363698 admin admin] RBAC: Authorizing `identity:delete_user(user_id=155450dfe7034865821a0472f1ba0966)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:09:28.813490 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef79984e-5611-45c4-9b80-3859fd363698 admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:09:28.868971 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-ef79984e-5611-45c4-9b80-3859fd363698 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '155450dfe7034865821a0472f1ba0966'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:09:28.877238 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-ef79984e-5611-45c4-9b80-3859fd363698 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '155450dfe7034865821a0472f1ba0966'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:09:28.892354 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-ef79984e-5611-45c4-9b80-3859fd363698 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '155450dfe7034865821a0472f1ba0966'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:09:28.893106 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-ef79984e-5611-45c4-9b80-3859fd363698 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '155450dfe7034865821a0472f1ba0966'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:09:28.901129 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-ef79984e-5611-45c4-9b80-3859fd363698 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '155450dfe7034865821a0472f1ba0966'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:09:28.907575 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1102/2240] 10.222.0.50 () {58 vars in 1224 bytes} [Tue Sep 13 16:09:28 2022] DELETE /identity/v3/users/155450dfe7034865821a0472f1ba0966 => generated 0 bytes in 172 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 13 16:09:28.912809 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce361028-a5de-423f-bbbf-1fd22d68e8ce None None] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:09:28.913322 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce361028-a5de-423f-bbbf-1fd22d68e8ce None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:09:28.913435 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce361028-a5de-423f-bbbf-1fd22d68e8ce None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:09:28.937141 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.auth.core [None req-ce361028-a5de-423f-bbbf-1fd22d68e8ce None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81437) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:09:28.938547 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-ce361028-a5de-423f-bbbf-1fd22d68e8ce None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:09:29.013448 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1139/2241] 10.222.0.50 () {58 vars in 958 bytes} [Tue Sep 13 16:09:28 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:09:29.023678 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef79984e-5611-45c4-9b80-3859fd363698 admin admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:09:29.025136 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-ef79984e-5611-45c4-9b80-3859fd363698 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:09:29.101622 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef79984e-5611-45c4-9b80-3859fd363698 admin admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:09:29.107497 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa654fc3-9f43-403f-a1aa-36abf0b29cf2 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f436f07939d4855a21b195bc7566244', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:09:29.108270 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa654fc3-9f43-403f-a1aa-36abf0b29cf2 service neutron] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:09:29.108648 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa654fc3-9f43-403f-a1aa-36abf0b29cf2 service neutron] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:09:29.109017 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa654fc3-9f43-403f-a1aa-36abf0b29cf2 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:09:29.121870 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa654fc3-9f43-403f-a1aa-36abf0b29cf2 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:09:29.123545 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa654fc3-9f43-403f-a1aa-36abf0b29cf2 service neutron] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:09:29.135227 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1103/2242] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 16:09:29 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 112 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 16:09:29.434509 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce361028-a5de-423f-bbbf-1fd22d68e8ce None None] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:09:29.436163 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-ce361028-a5de-423f-bbbf-1fd22d68e8ce None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:09:29.449514 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce361028-a5de-423f-bbbf-1fd22d68e8ce None None] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:09:29.454787 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bf4a660-9844-4ff4-92f0-0494c02b8037 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:09:29.455646 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bf4a660-9844-4ff4-92f0-0494c02b8037 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:09:29.456779 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bf4a660-9844-4ff4-92f0-0494c02b8037 admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:09:29.457064 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bf4a660-9844-4ff4-92f0-0494c02b8037 admin admin] PATH_INFO: `/v3/projects/991991dd8a1d4c4bb8c6fc4ea68ad9b8` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:09:29.458223 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2bf4a660-9844-4ff4-92f0-0494c02b8037 admin admin] RBAC: Authorizing `identity:delete_project(project_id=991991dd8a1d4c4bb8c6fc4ea68ad9b8)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:09:29.460606 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2bf4a660-9844-4ff4-92f0-0494c02b8037 admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:09:29.511315 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-2bf4a660-9844-4ff4-92f0-0494c02b8037 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '991991dd8a1d4c4bb8c6fc4ea68ad9b8', 'request_id': 'req-2bf4a660-9844-4ff4-92f0-0494c02b8037'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:09:29.520188 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-2bf4a660-9844-4ff4-92f0-0494c02b8037 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '991991dd8a1d4c4bb8c6fc4ea68ad9b8', 'request_id': 'req-2bf4a660-9844-4ff4-92f0-0494c02b8037'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:09:29.528350 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-2bf4a660-9844-4ff4-92f0-0494c02b8037 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '991991dd8a1d4c4bb8c6fc4ea68ad9b8', 'request_id': 'req-2bf4a660-9844-4ff4-92f0-0494c02b8037'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:09:29.540386 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-2bf4a660-9844-4ff4-92f0-0494c02b8037 admin admin] The token cache is being invalidate because project 991991dd8a1d4c4bb8c6fc4ea68ad9b8 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81437) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 13 16:09:29.540681 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-2bf4a660-9844-4ff4-92f0-0494c02b8037 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:09:29.541336 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-2bf4a660-9844-4ff4-92f0-0494c02b8037 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:09:29.542822 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1140/2243] 10.222.0.50 () {58 vars in 1233 bytes} [Tue Sep 13 16:09:29 2022] DELETE /identity/v3/projects/991991dd8a1d4c4bb8c6fc4ea68ad9b8 => generated 0 bytes in 109 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 16:09:29.547685 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29987864-59fc-487f-8717-3b584f8f7d34 None None] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:09:29.547970 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29987864-59fc-487f-8717-3b584f8f7d34 None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:09:29.548274 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29987864-59fc-487f-8717-3b584f8f7d34 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:09:29.571895 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.auth.core [None req-29987864-59fc-487f-8717-3b584f8f7d34 None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81438) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:09:29.573164 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-29987864-59fc-487f-8717-3b584f8f7d34 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:09:29.595221 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:09:29.654744 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1104/2244] 10.222.0.50 () {58 vars in 958 bytes} [Tue Sep 13 16:09:29 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:09:29.663251 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-719d5e29-a825-419b-94d0-c82fda7eea15 None None] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:09:29.663539 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-719d5e29-a825-419b-94d0-c82fda7eea15 None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:09:29.663818 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-719d5e29-a825-419b-94d0-c82fda7eea15 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:09:29.684376 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.auth.core [None req-719d5e29-a825-419b-94d0-c82fda7eea15 None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81437) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:09:29.685768 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-719d5e29-a825-419b-94d0-c82fda7eea15 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:09:29.693828 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1141/2245] 10.222.0.50 () {58 vars in 958 bytes} [Tue Sep 13 16:09:29 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:09:29.698210 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29987864-59fc-487f-8717-3b584f8f7d34 None None] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:09:29.710950 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29987864-59fc-487f-8717-3b584f8f7d34 None None] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:09:29.724131 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0858d129-e2fe-4987-8fdf-6b1fbeb7bd6a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:09:29.724707 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0858d129-e2fe-4987-8fdf-6b1fbeb7bd6a admin admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:09:29.725111 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0858d129-e2fe-4987-8fdf-6b1fbeb7bd6a admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:09:29.725397 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0858d129-e2fe-4987-8fdf-6b1fbeb7bd6a admin admin] PATH_INFO: `/v3/domains` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:09:29.726061 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0858d129-e2fe-4987-8fdf-6b1fbeb7bd6a admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81438) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 16:09:29.726423 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0858d129-e2fe-4987-8fdf-6b1fbeb7bd6a admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:09:29.728375 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0858d129-e2fe-4987-8fdf-6b1fbeb7bd6a admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:09:29.744370 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1105/2246] 10.222.0.50 () {58 vars in 1153 bytes} [Tue Sep 13 16:09:29 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 47 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 16:09:29.748820 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-719d5e29-a825-419b-94d0-c82fda7eea15 None None] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:09:29.770072 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-719d5e29-a825-419b-94d0-c82fda7eea15 None None] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:09:29.770072 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c0c8e84-fde6-4be1-8137-c720f25d548b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:09:29.770072 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c0c8e84-fde6-4be1-8137-c720f25d548b admin admin] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:09:29.770072 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c0c8e84-fde6-4be1-8137-c720f25d548b admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:09:29.770448 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c0c8e84-fde6-4be1-8137-c720f25d548b admin admin] PATH_INFO: `/v3/projects` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:09:29.770550 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c0c8e84-fde6-4be1-8137-c720f25d548b admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:09:29.772610 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c0c8e84-fde6-4be1-8137-c720f25d548b admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:09:29.844194 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1142/2247] 10.222.0.50 () {60 vars in 1153 bytes} [Tue Sep 13 16:09:29 2022] POST /identity/v3/projects => generated 375 bytes in 96 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 16:09:29.849271 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0858d129-e2fe-4987-8fdf-6b1fbeb7bd6a admin admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:09:29.859369 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0858d129-e2fe-4987-8fdf-6b1fbeb7bd6a admin admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:09:29.934383 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4719593d-73c5-49ba-ac33-3b68f763d376 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:09:29.935050 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4719593d-73c5-49ba-ac33-3b68f763d376 admin admin] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:09:29.935339 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4719593d-73c5-49ba-ac33-3b68f763d376 admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:09:29.935619 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4719593d-73c5-49ba-ac33-3b68f763d376 admin admin] PATH_INFO: `/v3/users` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:09:29.936282 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4719593d-73c5-49ba-ac33-3b68f763d376 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:09:29.938371 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4719593d-73c5-49ba-ac33-3b68f763d376 admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:09:29.974349 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1106/2248] 10.222.0.50 () {60 vars in 1144 bytes} [Tue Sep 13 16:09:29 2022] POST /identity/v3/users => generated 288 bytes in 127 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 13 16:09:29.978303 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c0c8e84-fde6-4be1-8137-c720f25d548b admin admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:09:29.989403 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c0c8e84-fde6-4be1-8137-c720f25d548b admin admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:09:29.994056 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-126bb4bd-a7bc-49a4-96c4-19fbb0840c4d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:09:29.994655 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-126bb4bd-a7bc-49a4-96c4-19fbb0840c4d admin admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:09:29.995031 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-126bb4bd-a7bc-49a4-96c4-19fbb0840c4d admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:09:29.995314 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-126bb4bd-a7bc-49a4-96c4-19fbb0840c4d admin admin] PATH_INFO: `/v3/roles` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:09:29.995955 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-126bb4bd-a7bc-49a4-96c4-19fbb0840c4d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:09:29.997547 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-126bb4bd-a7bc-49a4-96c4-19fbb0840c4d admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:09:30.011207 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1143/2249] 10.222.0.50 () {58 vars in 1122 bytes} [Tue Sep 13 16:09:29 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 16:09:30.015090 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4719593d-73c5-49ba-ac33-3b68f763d376 admin admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:09:30.026200 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4719593d-73c5-49ba-ac33-3b68f763d376 admin admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:09:30.030930 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72889f7e-f0fd-433a-a2af-f5f60075d1e7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:09:30.031550 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72889f7e-f0fd-433a-a2af-f5f60075d1e7 admin admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:09:30.031831 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72889f7e-f0fd-433a-a2af-f5f60075d1e7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:09:30.032113 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72889f7e-f0fd-433a-a2af-f5f60075d1e7 admin admin] PATH_INFO: `/v3/roles` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:09:30.032752 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72889f7e-f0fd-433a-a2af-f5f60075d1e7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:09:30.034451 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72889f7e-f0fd-433a-a2af-f5f60075d1e7 admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:09:30.047522 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1107/2250] 10.222.0.50 () {58 vars in 1122 bytes} [Tue Sep 13 16:09:30 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 16:09:30.051847 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-126bb4bd-a7bc-49a4-96c4-19fbb0840c4d admin admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:09:30.062142 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-126bb4bd-a7bc-49a4-96c4-19fbb0840c4d admin admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:09:30.066838 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e92a3147-ddd9-43c2-b84e-f9b9311351f1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:09:30.067467 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e92a3147-ddd9-43c2-b84e-f9b9311351f1 admin admin] REQUEST_METHOD: `PUT` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:09:30.067789 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e92a3147-ddd9-43c2-b84e-f9b9311351f1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:09:30.068063 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e92a3147-ddd9-43c2-b84e-f9b9311351f1 admin admin] PATH_INFO: `/v3/projects/e33139c4cfa44c479f47c277c069db84/users/3e4d2ae878b346469f6a8ab751a7055d/roles/15db4aaca3264ae29bde38903ba43471` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:09:30.084510 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e92a3147-ddd9-43c2-b84e-f9b9311351f1 admin admin] RBAC: Authorizing `identity:create_grant(project_id=e33139c4cfa44c479f47c277c069db84, user_id=3e4d2ae878b346469f6a8ab751a7055d, role_id=15db4aaca3264ae29bde38903ba43471)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:09:30.086937 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e92a3147-ddd9-43c2-b84e-f9b9311351f1 admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:09:30.105041 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1144/2251] 10.222.0.50 () {60 vars in 1483 bytes} [Tue Sep 13 16:09:30 2022] PUT /identity/v3/projects/e33139c4cfa44c479f47c277c069db84/users/3e4d2ae878b346469f6a8ab751a7055d/roles/15db4aaca3264ae29bde38903ba43471 => generated 0 bytes in 54 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 16:09:30.116575 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28717463-5044-4e0e-bddd-0ed89544af57 None None] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:09:30.116865 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28717463-5044-4e0e-bddd-0ed89544af57 None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:09:30.117267 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28717463-5044-4e0e-bddd-0ed89544af57 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:09:30.146352 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.auth.core [None req-28717463-5044-4e0e-bddd-0ed89544af57 None None] MFA Rules not processed for user `3e4d2ae878b346469f6a8ab751a7055d`. Rule list: `[]` (Enabled: `True`). {{(pid=81438) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:09:30.147598 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-28717463-5044-4e0e-bddd-0ed89544af57 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:09:30.224934 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1108/2252] 10.222.0.50 () {58 vars in 958 bytes} [Tue Sep 13 16:09:30 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:09:30.235849 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e92a3147-ddd9-43c2-b84e-f9b9311351f1 admin admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:09:30.237655 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-e92a3147-ddd9-43c2-b84e-f9b9311351f1 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:09:30.260749 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:09:30.300944 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e92a3147-ddd9-43c2-b84e-f9b9311351f1 admin admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:09:30.304680 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df85b6fc-7cc2-4fec-bfff-c9a17666f7bf service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7fd6b5afd67b48f9b0f0597ccd4cf4df', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:09:30.305277 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df85b6fc-7cc2-4fec-bfff-c9a17666f7bf service cinder] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:09:30.305553 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df85b6fc-7cc2-4fec-bfff-c9a17666f7bf service cinder] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:09:30.305823 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df85b6fc-7cc2-4fec-bfff-c9a17666f7bf service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:09:30.314174 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df85b6fc-7cc2-4fec-bfff-c9a17666f7bf service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:09:30.315713 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df85b6fc-7cc2-4fec-bfff-c9a17666f7bf service cinder] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:09:30.325069 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1145/2253] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 16:09:30 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 90 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 16:09:33.513871 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28717463-5044-4e0e-bddd-0ed89544af57 None None] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:09:33.557628 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:09:33.620920 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28717463-5044-4e0e-bddd-0ed89544af57 None None] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:09:33.626090 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd70072a-4f1e-4691-9d29-14810206d346 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:09:33.626602 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd70072a-4f1e-4691-9d29-14810206d346 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:09:33.626891 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd70072a-4f1e-4691-9d29-14810206d346 admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:09:33.627191 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd70072a-4f1e-4691-9d29-14810206d346 admin admin] PATH_INFO: `/v3/users/3e4d2ae878b346469f6a8ab751a7055d` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:09:33.628331 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd70072a-4f1e-4691-9d29-14810206d346 admin admin] RBAC: Authorizing `identity:delete_user(user_id=3e4d2ae878b346469f6a8ab751a7055d)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:09:33.630615 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd70072a-4f1e-4691-9d29-14810206d346 admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:09:33.718349 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-bd70072a-4f1e-4691-9d29-14810206d346 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '3e4d2ae878b346469f6a8ab751a7055d'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:09:33.729341 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-bd70072a-4f1e-4691-9d29-14810206d346 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '3e4d2ae878b346469f6a8ab751a7055d'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:09:33.746315 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-bd70072a-4f1e-4691-9d29-14810206d346 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '3e4d2ae878b346469f6a8ab751a7055d'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:09:33.747414 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-bd70072a-4f1e-4691-9d29-14810206d346 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '3e4d2ae878b346469f6a8ab751a7055d'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:09:33.757500 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-bd70072a-4f1e-4691-9d29-14810206d346 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '3e4d2ae878b346469f6a8ab751a7055d'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:09:33.771015 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1109/2254] 10.222.0.50 () {58 vars in 1224 bytes} [Tue Sep 13 16:09:33 2022] DELETE /identity/v3/users/3e4d2ae878b346469f6a8ab751a7055d => generated 0 bytes in 251 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 16:09:33.771497 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f54d7d6-940e-4381-a430-32e1a920fc2d None None] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:09:33.771497 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f54d7d6-940e-4381-a430-32e1a920fc2d None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:09:33.771497 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f54d7d6-940e-4381-a430-32e1a920fc2d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:09:33.796558 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.auth.core [None req-0f54d7d6-940e-4381-a430-32e1a920fc2d None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81437) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:09:33.797684 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-0f54d7d6-940e-4381-a430-32e1a920fc2d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:09:33.891123 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1146/2255] 10.222.0.50 () {58 vars in 958 bytes} [Tue Sep 13 16:09:33 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 124 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:09:33.902050 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd70072a-4f1e-4691-9d29-14810206d346 admin admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:09:33.903546 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-bd70072a-4f1e-4691-9d29-14810206d346 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:09:33.973221 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd70072a-4f1e-4691-9d29-14810206d346 admin admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:09:33.977588 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09dd3be8-5838-470e-af9b-2b960deba842 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f436f07939d4855a21b195bc7566244', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:09:33.978159 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09dd3be8-5838-470e-af9b-2b960deba842 service neutron] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:09:33.978437 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09dd3be8-5838-470e-af9b-2b960deba842 service neutron] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:09:33.978869 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09dd3be8-5838-470e-af9b-2b960deba842 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:09:33.990167 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09dd3be8-5838-470e-af9b-2b960deba842 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:09:33.991441 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09dd3be8-5838-470e-af9b-2b960deba842 service neutron] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:09:34.011808 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1110/2256] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 16:09:33 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 110 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 16:09:34.337051 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f54d7d6-940e-4381-a430-32e1a920fc2d None None] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:09:34.339273 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-0f54d7d6-940e-4381-a430-32e1a920fc2d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:09:34.365447 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f54d7d6-940e-4381-a430-32e1a920fc2d None None] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:09:34.371886 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc6e7255-8db0-4fea-81b4-1fa5db4e701d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:09:34.372642 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc6e7255-8db0-4fea-81b4-1fa5db4e701d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:09:34.372991 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc6e7255-8db0-4fea-81b4-1fa5db4e701d admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:09:34.373435 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc6e7255-8db0-4fea-81b4-1fa5db4e701d admin admin] PATH_INFO: `/v3/projects/e33139c4cfa44c479f47c277c069db84` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:09:34.374572 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc6e7255-8db0-4fea-81b4-1fa5db4e701d admin admin] RBAC: Authorizing `identity:delete_project(project_id=e33139c4cfa44c479f47c277c069db84)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:09:34.376857 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc6e7255-8db0-4fea-81b4-1fa5db4e701d admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:09:34.429092 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-cc6e7255-8db0-4fea-81b4-1fa5db4e701d admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'e33139c4cfa44c479f47c277c069db84', 'request_id': 'req-cc6e7255-8db0-4fea-81b4-1fa5db4e701d'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:09:34.436893 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-cc6e7255-8db0-4fea-81b4-1fa5db4e701d admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'e33139c4cfa44c479f47c277c069db84', 'request_id': 'req-cc6e7255-8db0-4fea-81b4-1fa5db4e701d'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:09:34.445051 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-cc6e7255-8db0-4fea-81b4-1fa5db4e701d admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'e33139c4cfa44c479f47c277c069db84', 'request_id': 'req-cc6e7255-8db0-4fea-81b4-1fa5db4e701d'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:09:34.455832 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-cc6e7255-8db0-4fea-81b4-1fa5db4e701d admin admin] The token cache is being invalidate because project e33139c4cfa44c479f47c277c069db84 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81437) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 13 16:09:34.456124 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-cc6e7255-8db0-4fea-81b4-1fa5db4e701d admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:09:34.457341 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-cc6e7255-8db0-4fea-81b4-1fa5db4e701d admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:09:34.459427 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1147/2257] 10.222.0.50 () {58 vars in 1233 bytes} [Tue Sep 13 16:09:34 2022] DELETE /identity/v3/projects/e33139c4cfa44c479f47c277c069db84 => generated 0 bytes in 124 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 16:09:34.464362 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20ec873d-6fa8-4fdf-a99f-8c84dd67a14b None None] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:09:34.464651 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20ec873d-6fa8-4fdf-a99f-8c84dd67a14b None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:09:34.464932 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20ec873d-6fa8-4fdf-a99f-8c84dd67a14b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:09:34.486227 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.auth.core [None req-20ec873d-6fa8-4fdf-a99f-8c84dd67a14b None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81438) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:09:34.487357 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-20ec873d-6fa8-4fdf-a99f-8c84dd67a14b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:09:34.525017 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:09:34.563856 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1111/2258] 10.222.0.50 () {58 vars in 958 bytes} [Tue Sep 13 16:09:34 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:09:34.572545 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9de3bad-54d2-44c9-b90b-6f570749157e None None] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:09:34.573157 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9de3bad-54d2-44c9-b90b-6f570749157e None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:09:34.573646 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9de3bad-54d2-44c9-b90b-6f570749157e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:09:34.574348 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca01fc2b-e0aa-4ded-97d3-175d725ebbbc None None] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:09:34.574637 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca01fc2b-e0aa-4ded-97d3-175d725ebbbc None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:09:34.575027 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca01fc2b-e0aa-4ded-97d3-175d725ebbbc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:09:34.596645 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.auth.core [None req-ca01fc2b-e0aa-4ded-97d3-175d725ebbbc None None] MFA Rules not processed for user `f8364b1a190d4218bf353a89bdb81448`. Rule list: `[]` (Enabled: `True`). {{(pid=81438) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:09:34.598104 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.auth.core [None req-a9de3bad-54d2-44c9-b90b-6f570749157e None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81437) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:09:34.598660 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-ca01fc2b-e0aa-4ded-97d3-175d725ebbbc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:09:34.599908 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-a9de3bad-54d2-44c9-b90b-6f570749157e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:09:34.608053 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1148/2259] 10.222.0.50 () {58 vars in 958 bytes} [Tue Sep 13 16:09:34 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 37 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:09:34.613504 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9de3bad-54d2-44c9-b90b-6f570749157e None None] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:09:34.626542 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9de3bad-54d2-44c9-b90b-6f570749157e None None] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:09:34.633700 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50866faf-d716-4b7c-86cb-833ca1c6fbea admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:09:34.634320 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50866faf-d716-4b7c-86cb-833ca1c6fbea admin admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:09:34.634604 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50866faf-d716-4b7c-86cb-833ca1c6fbea admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:09:34.634913 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50866faf-d716-4b7c-86cb-833ca1c6fbea admin admin] PATH_INFO: `/v3/domains` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:09:34.635559 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50866faf-d716-4b7c-86cb-833ca1c6fbea admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81437) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 16:09:34.635981 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50866faf-d716-4b7c-86cb-833ca1c6fbea admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:09:34.637620 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50866faf-d716-4b7c-86cb-833ca1c6fbea admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:09:34.650185 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1149/2260] 10.222.0.50 () {58 vars in 1153 bytes} [Tue Sep 13 16:09:34 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 13 16:09:34.655139 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50866faf-d716-4b7c-86cb-833ca1c6fbea admin admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:09:34.667064 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50866faf-d716-4b7c-86cb-833ca1c6fbea admin admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:09:34.672954 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e77b24b-d176-48ca-bb06-451fa6df363a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:09:34.673555 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e77b24b-d176-48ca-bb06-451fa6df363a admin admin] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:09:34.674021 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e77b24b-d176-48ca-bb06-451fa6df363a admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:09:34.674165 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e77b24b-d176-48ca-bb06-451fa6df363a admin admin] PATH_INFO: `/v3/projects` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:09:34.674829 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e77b24b-d176-48ca-bb06-451fa6df363a admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:09:34.677431 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e77b24b-d176-48ca-bb06-451fa6df363a admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:09:34.680221 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1112/2261] 10.222.0.50 () {58 vars in 1009 bytes} [Tue Sep 13 16:09:34 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 13 16:09:34.717210 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1150/2262] 10.222.0.50 () {60 vars in 1153 bytes} [Tue Sep 13 16:09:34 2022] POST /identity/v3/projects => generated 383 bytes in 60 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 16:09:34.719530 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca01fc2b-e0aa-4ded-97d3-175d725ebbbc None None] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:09:34.739723 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca01fc2b-e0aa-4ded-97d3-175d725ebbbc None None] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:09:34.752655 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e77b24b-d176-48ca-bb06-451fa6df363a admin admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:09:34.755283 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-6e77b24b-d176-48ca-bb06-451fa6df363a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:09:34.762586 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:09:34.853062 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f530cac2-6b51-4442-bd29-04263109c627 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:09:34.853778 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f530cac2-6b51-4442-bd29-04263109c627 admin admin] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:09:34.854121 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f530cac2-6b51-4442-bd29-04263109c627 admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:09:34.854459 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f530cac2-6b51-4442-bd29-04263109c627 admin admin] PATH_INFO: `/v3/users` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:09:34.855260 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f530cac2-6b51-4442-bd29-04263109c627 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:09:34.857358 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f530cac2-6b51-4442-bd29-04263109c627 admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:09:34.868446 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e77b24b-d176-48ca-bb06-451fa6df363a admin admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:09:34.879208 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-889cea6e-e5b1-4589-95dc-412f6a261f52 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b00ab0d6321d483188bc69deaddcdbbc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:09:34.880248 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-889cea6e-e5b1-4589-95dc-412f6a261f52 service swift] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:09:34.880501 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-889cea6e-e5b1-4589-95dc-412f6a261f52 service swift] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:09:34.880856 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-889cea6e-e5b1-4589-95dc-412f6a261f52 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:09:34.891122 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1113/2263] 10.222.0.50 () {60 vars in 1144 bytes} [Tue Sep 13 16:09:34 2022] POST /identity/v3/users => generated 292 bytes in 173 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 16:09:34.903159 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f530cac2-6b51-4442-bd29-04263109c627 admin admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:09:34.919388 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f530cac2-6b51-4442-bd29-04263109c627 admin admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:09:34.928008 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae47f859-9cf0-4e6a-b367-d93eacd21486 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:09:34.928912 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae47f859-9cf0-4e6a-b367-d93eacd21486 admin admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:09:34.929092 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae47f859-9cf0-4e6a-b367-d93eacd21486 admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:09:34.929491 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae47f859-9cf0-4e6a-b367-d93eacd21486 admin admin] PATH_INFO: `/v3/roles` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:09:34.930158 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae47f859-9cf0-4e6a-b367-d93eacd21486 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:09:34.932196 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae47f859-9cf0-4e6a-b367-d93eacd21486 admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:09:34.945707 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1114/2264] 10.222.0.50 () {58 vars in 1122 bytes} [Tue Sep 13 16:09:34 2022] GET /identity/v3/roles => generated 1420 bytes in 43 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 16:09:34.955940 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae47f859-9cf0-4e6a-b367-d93eacd21486 admin admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:09:34.966682 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae47f859-9cf0-4e6a-b367-d93eacd21486 admin admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:09:34.969429 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-889cea6e-e5b1-4589-95dc-412f6a261f52 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:09:34.972737 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eefd003f-e3b2-458e-a2a4-6acea0f0ccde admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:09:34.973968 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eefd003f-e3b2-458e-a2a4-6acea0f0ccde admin admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:09:34.974388 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eefd003f-e3b2-458e-a2a4-6acea0f0ccde admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:09:34.974722 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eefd003f-e3b2-458e-a2a4-6acea0f0ccde admin admin] PATH_INFO: `/v3/roles` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:09:34.975492 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eefd003f-e3b2-458e-a2a4-6acea0f0ccde admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:09:34.977393 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eefd003f-e3b2-458e-a2a4-6acea0f0ccde admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:09:34.978741 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-889cea6e-e5b1-4589-95dc-412f6a261f52 service swift] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:09:34.993965 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1115/2265] 10.222.0.50 () {58 vars in 1122 bytes} [Tue Sep 13 16:09:34 2022] GET /identity/v3/roles => generated 1420 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 16:09:34.997534 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1151/2266] 10.222.0.50 () {60 vars in 1384 bytes} [Tue Sep 13 16:09:34 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 245 msecs (HTTP/1.1 200) 6 headers in 379 bytes (2 switches on core 0) Sep 13 16:09:34.998281 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eefd003f-e3b2-458e-a2a4-6acea0f0ccde admin admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:09:35.011809 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eefd003f-e3b2-458e-a2a4-6acea0f0ccde admin admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:09:35.023269 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6557c02-2d33-4df9-b469-fb4469ded414 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:09:35.024054 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6557c02-2d33-4df9-b469-fb4469ded414 admin admin] REQUEST_METHOD: `PUT` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:09:35.024323 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6557c02-2d33-4df9-b469-fb4469ded414 admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:09:35.024740 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6557c02-2d33-4df9-b469-fb4469ded414 admin admin] PATH_INFO: `/v3/projects/e9ad96d64d5c431098e52f0bd8c0473c/users/756e733b0cf14173aeb0f85b44bfbfaf/roles/15db4aaca3264ae29bde38903ba43471` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:09:35.071322 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6557c02-2d33-4df9-b469-fb4469ded414 admin admin] RBAC: Authorizing `identity:create_grant(project_id=e9ad96d64d5c431098e52f0bd8c0473c, user_id=756e733b0cf14173aeb0f85b44bfbfaf, role_id=15db4aaca3264ae29bde38903ba43471)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:09:35.074658 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6557c02-2d33-4df9-b469-fb4469ded414 admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:09:35.106100 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1116/2267] 10.222.0.50 () {60 vars in 1483 bytes} [Tue Sep 13 16:09:34 2022] PUT /identity/v3/projects/e9ad96d64d5c431098e52f0bd8c0473c/users/756e733b0cf14173aeb0f85b44bfbfaf/roles/15db4aaca3264ae29bde38903ba43471 => generated 0 bytes in 99 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 13 16:09:35.108773 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94b493ca-722b-42f8-a855-b902a3144616 None None] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:09:35.109231 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94b493ca-722b-42f8-a855-b902a3144616 None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:09:35.109683 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94b493ca-722b-42f8-a855-b902a3144616 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:09:35.141142 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.auth.core [None req-94b493ca-722b-42f8-a855-b902a3144616 None None] MFA Rules not processed for user `756e733b0cf14173aeb0f85b44bfbfaf`. Rule list: `[]` (Enabled: `True`). {{(pid=81437) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:09:35.142669 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-94b493ca-722b-42f8-a855-b902a3144616 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:09:35.228229 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1152/2268] 10.222.0.50 () {58 vars in 958 bytes} [Tue Sep 13 16:09:35 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 120 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:09:35.229025 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6557c02-2d33-4df9-b469-fb4469ded414 admin admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:09:35.267096 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:09:35.307998 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6557c02-2d33-4df9-b469-fb4469ded414 admin admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:09:35.312952 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d5870c8-63f7-4204-9e91-b7a2d90c4516 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:09:35.313822 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d5870c8-63f7-4204-9e91-b7a2d90c4516 admin admin] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:09:35.314214 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d5870c8-63f7-4204-9e91-b7a2d90c4516 admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:09:35.314622 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d5870c8-63f7-4204-9e91-b7a2d90c4516 admin admin] PATH_INFO: `/v3/projects` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:09:35.315411 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d5870c8-63f7-4204-9e91-b7a2d90c4516 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:09:35.317887 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d5870c8-63f7-4204-9e91-b7a2d90c4516 admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:09:35.363514 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1117/2269] 10.222.0.50 () {60 vars in 1153 bytes} [Tue Sep 13 16:09:35 2022] POST /identity/v3/projects => generated 383 bytes in 132 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 13 16:09:35.365058 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94b493ca-722b-42f8-a855-b902a3144616 None None] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:09:35.377747 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94b493ca-722b-42f8-a855-b902a3144616 None None] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:09:35.415301 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:09:35.446406 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50677c7f-51b5-4624-88b0-9dc7bd1aff77 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:09:35.447423 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50677c7f-51b5-4624-88b0-9dc7bd1aff77 admin admin] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:09:35.447863 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50677c7f-51b5-4624-88b0-9dc7bd1aff77 admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:09:35.448292 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50677c7f-51b5-4624-88b0-9dc7bd1aff77 admin admin] PATH_INFO: `/v3/users` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:09:35.449118 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50677c7f-51b5-4624-88b0-9dc7bd1aff77 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:09:35.451190 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50677c7f-51b5-4624-88b0-9dc7bd1aff77 admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:09:35.475607 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1153/2270] 10.222.0.50 () {60 vars in 1144 bytes} [Tue Sep 13 16:09:35 2022] POST /identity/v3/users => generated 292 bytes in 111 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 16:09:35.479969 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d5870c8-63f7-4204-9e91-b7a2d90c4516 admin admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:09:35.497987 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d5870c8-63f7-4204-9e91-b7a2d90c4516 admin admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:09:35.505821 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41c4e05d-d51d-443c-a232-c2be1c566a5c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:09:35.505821 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41c4e05d-d51d-443c-a232-c2be1c566a5c admin admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:09:35.505821 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41c4e05d-d51d-443c-a232-c2be1c566a5c admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:09:35.505821 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41c4e05d-d51d-443c-a232-c2be1c566a5c admin admin] PATH_INFO: `/v3/roles` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:09:35.506252 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41c4e05d-d51d-443c-a232-c2be1c566a5c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:09:35.507256 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41c4e05d-d51d-443c-a232-c2be1c566a5c admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:09:35.521859 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1118/2271] 10.222.0.50 () {58 vars in 1122 bytes} [Tue Sep 13 16:09:35 2022] GET /identity/v3/roles => generated 1420 bytes in 40 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 13 16:09:35.523033 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50677c7f-51b5-4624-88b0-9dc7bd1aff77 admin admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:09:35.533370 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50677c7f-51b5-4624-88b0-9dc7bd1aff77 admin admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:09:35.538698 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0f7f394-fccc-4577-812f-cafa6c080f25 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:09:35.539350 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0f7f394-fccc-4577-812f-cafa6c080f25 admin admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:09:35.539628 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0f7f394-fccc-4577-812f-cafa6c080f25 admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:09:35.539905 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0f7f394-fccc-4577-812f-cafa6c080f25 admin admin] PATH_INFO: `/v3/roles` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:09:35.540555 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0f7f394-fccc-4577-812f-cafa6c080f25 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:09:35.542190 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0f7f394-fccc-4577-812f-cafa6c080f25 admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:09:35.554424 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1154/2272] 10.222.0.50 () {58 vars in 1122 bytes} [Tue Sep 13 16:09:35 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 13 16:09:35.558444 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41c4e05d-d51d-443c-a232-c2be1c566a5c admin admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:09:35.570378 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41c4e05d-d51d-443c-a232-c2be1c566a5c admin admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:09:35.575482 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ffbdea5-d795-4f07-b5a7-32557948d9fa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:09:35.576245 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ffbdea5-d795-4f07-b5a7-32557948d9fa admin admin] REQUEST_METHOD: `PUT` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:09:35.576596 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ffbdea5-d795-4f07-b5a7-32557948d9fa admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:09:35.576939 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ffbdea5-d795-4f07-b5a7-32557948d9fa admin admin] PATH_INFO: `/v3/projects/b3bfb0ffd8244b2c840847473ff3c993/users/1598727725b3437b9a8f36471293f677/roles/15db4aaca3264ae29bde38903ba43471` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:09:35.593339 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ffbdea5-d795-4f07-b5a7-32557948d9fa admin admin] RBAC: Authorizing `identity:create_grant(project_id=b3bfb0ffd8244b2c840847473ff3c993, user_id=1598727725b3437b9a8f36471293f677, role_id=15db4aaca3264ae29bde38903ba43471)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:09:35.596183 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ffbdea5-d795-4f07-b5a7-32557948d9fa admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:09:35.613392 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1119/2273] 10.222.0.50 () {60 vars in 1483 bytes} [Tue Sep 13 16:09:35 2022] PUT /identity/v3/projects/b3bfb0ffd8244b2c840847473ff3c993/users/1598727725b3437b9a8f36471293f677/roles/15db4aaca3264ae29bde38903ba43471 => generated 0 bytes in 55 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 16:09:35.690642 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bbdbece-10ad-40b1-a6d0-4af47ce42769 None None] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:09:35.691149 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bbdbece-10ad-40b1-a6d0-4af47ce42769 None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:09:35.691478 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bbdbece-10ad-40b1-a6d0-4af47ce42769 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:09:35.723073 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.auth.core [None req-5bbdbece-10ad-40b1-a6d0-4af47ce42769 None None] MFA Rules not processed for user `1598727725b3437b9a8f36471293f677`. Rule list: `[]` (Enabled: `True`). {{(pid=81437) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:09:35.724325 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-5bbdbece-10ad-40b1-a6d0-4af47ce42769 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:09:35.807631 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1155/2274] 10.222.0.50 () {58 vars in 958 bytes} [Tue Sep 13 16:09:35 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 118 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:09:35.812998 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ffbdea5-d795-4f07-b5a7-32557948d9fa admin admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:09:35.897995 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ffbdea5-d795-4f07-b5a7-32557948d9fa admin admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:09:35.902977 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de0ba3b0-5099-4857-9945-c498f2e43edc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:09:35.903622 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de0ba3b0-5099-4857-9945-c498f2e43edc admin admin] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:09:35.903900 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de0ba3b0-5099-4857-9945-c498f2e43edc admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:09:35.904168 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de0ba3b0-5099-4857-9945-c498f2e43edc admin admin] PATH_INFO: `/v3/projects` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:09:35.904810 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de0ba3b0-5099-4857-9945-c498f2e43edc admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:09:35.906735 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de0ba3b0-5099-4857-9945-c498f2e43edc admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:09:35.939426 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1120/2275] 10.222.0.50 () {60 vars in 1153 bytes} [Tue Sep 13 16:09:35 2022] POST /identity/v3/projects => generated 385 bytes in 127 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 16:09:35.944166 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bbdbece-10ad-40b1-a6d0-4af47ce42769 None None] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:09:35.960293 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bbdbece-10ad-40b1-a6d0-4af47ce42769 None None] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:09:36.115380 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b5144d1-d70d-499e-b3cd-0a6210bba8c2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:09:36.116007 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b5144d1-d70d-499e-b3cd-0a6210bba8c2 admin admin] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:09:36.116287 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b5144d1-d70d-499e-b3cd-0a6210bba8c2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:09:36.116562 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b5144d1-d70d-499e-b3cd-0a6210bba8c2 admin admin] PATH_INFO: `/v3/users` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:09:36.117202 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b5144d1-d70d-499e-b3cd-0a6210bba8c2 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:09:36.119164 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b5144d1-d70d-499e-b3cd-0a6210bba8c2 admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:09:36.177467 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1156/2276] 10.222.0.50 () {60 vars in 1144 bytes} [Tue Sep 13 16:09:35 2022] POST /identity/v3/users => generated 299 bytes in 234 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 16:09:36.189465 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de0ba3b0-5099-4857-9945-c498f2e43edc admin admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:09:36.200997 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de0ba3b0-5099-4857-9945-c498f2e43edc admin admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:09:36.205797 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38bcd0a9-8f24-4548-9ce4-5f3b5d69c6d1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:09:36.207898 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38bcd0a9-8f24-4548-9ce4-5f3b5d69c6d1 admin admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:09:36.208317 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38bcd0a9-8f24-4548-9ce4-5f3b5d69c6d1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:09:36.208740 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38bcd0a9-8f24-4548-9ce4-5f3b5d69c6d1 admin admin] PATH_INFO: `/v3/roles` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:09:36.209678 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38bcd0a9-8f24-4548-9ce4-5f3b5d69c6d1 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:09:36.211817 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38bcd0a9-8f24-4548-9ce4-5f3b5d69c6d1 admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:09:36.226812 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1121/2277] 10.222.0.50 () {58 vars in 1122 bytes} [Tue Sep 13 16:09:36 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 16:09:36.228231 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b5144d1-d70d-499e-b3cd-0a6210bba8c2 admin admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:09:36.246591 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b5144d1-d70d-499e-b3cd-0a6210bba8c2 admin admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:09:36.261033 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6169800-0a53-473a-bdb0-2ad3e10d2a94 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:09:36.262023 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6169800-0a53-473a-bdb0-2ad3e10d2a94 admin admin] REQUEST_METHOD: `PUT` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:09:36.262312 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6169800-0a53-473a-bdb0-2ad3e10d2a94 admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:09:36.262737 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6169800-0a53-473a-bdb0-2ad3e10d2a94 admin admin] PATH_INFO: `/v3/projects/387d492fa25d49ef867c67942c2a7c68/users/e1eaf25bc6ce4ce1b8ac9da3a6322f8d/roles/031d4e1a23e34f71858fc4a6dfbd4ccb` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:09:36.295834 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6169800-0a53-473a-bdb0-2ad3e10d2a94 admin admin] RBAC: Authorizing `identity:create_grant(project_id=387d492fa25d49ef867c67942c2a7c68, user_id=e1eaf25bc6ce4ce1b8ac9da3a6322f8d, role_id=031d4e1a23e34f71858fc4a6dfbd4ccb)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:09:36.299089 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6169800-0a53-473a-bdb0-2ad3e10d2a94 admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:09:36.317445 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1157/2278] 10.222.0.50 () {60 vars in 1483 bytes} [Tue Sep 13 16:09:36 2022] PUT /identity/v3/projects/387d492fa25d49ef867c67942c2a7c68/users/e1eaf25bc6ce4ce1b8ac9da3a6322f8d/roles/031d4e1a23e34f71858fc4a6dfbd4ccb => generated 0 bytes in 90 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 16:09:36.321543 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38bcd0a9-8f24-4548-9ce4-5f3b5d69c6d1 admin admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:09:36.332509 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38bcd0a9-8f24-4548-9ce4-5f3b5d69c6d1 admin admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:09:36.395823 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:09:36.408121 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42f901a7-843f-4a78-8755-5b1ef1ab2731 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:09:36.408972 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42f901a7-843f-4a78-8755-5b1ef1ab2731 admin admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:09:36.411004 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42f901a7-843f-4a78-8755-5b1ef1ab2731 admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:09:36.411004 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42f901a7-843f-4a78-8755-5b1ef1ab2731 admin admin] PATH_INFO: `/v3/roles` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:09:36.411004 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42f901a7-843f-4a78-8755-5b1ef1ab2731 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:09:36.413564 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42f901a7-843f-4a78-8755-5b1ef1ab2731 admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:09:36.426949 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1122/2279] 10.222.0.50 () {58 vars in 1122 bytes} [Tue Sep 13 16:09:36 2022] GET /identity/v3/roles => generated 1420 bytes in 106 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 13 16:09:36.430972 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6169800-0a53-473a-bdb0-2ad3e10d2a94 admin admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:09:36.442921 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6169800-0a53-473a-bdb0-2ad3e10d2a94 admin admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:09:36.447768 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99c11cab-e462-45d5-b926-86ee9a8b742d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:09:36.448421 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99c11cab-e462-45d5-b926-86ee9a8b742d admin admin] REQUEST_METHOD: `PUT` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:09:36.448808 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99c11cab-e462-45d5-b926-86ee9a8b742d admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:09:36.449089 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99c11cab-e462-45d5-b926-86ee9a8b742d admin admin] PATH_INFO: `/v3/projects/387d492fa25d49ef867c67942c2a7c68/users/e1eaf25bc6ce4ce1b8ac9da3a6322f8d/roles/031d4e1a23e34f71858fc4a6dfbd4ccb` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:09:36.451038 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99c11cab-e462-45d5-b926-86ee9a8b742d admin admin] RBAC: Authorizing `identity:create_grant(project_id=387d492fa25d49ef867c67942c2a7c68, user_id=e1eaf25bc6ce4ce1b8ac9da3a6322f8d, role_id=031d4e1a23e34f71858fc4a6dfbd4ccb)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:09:36.453694 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99c11cab-e462-45d5-b926-86ee9a8b742d admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:09:36.457307 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=92,keystone:INSERT=8 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:09:36.468530 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1158/2280] 10.222.0.50 () {60 vars in 1483 bytes} [Tue Sep 13 16:09:36 2022] PUT /identity/v3/projects/387d492fa25d49ef867c67942c2a7c68/users/e1eaf25bc6ce4ce1b8ac9da3a6322f8d/roles/031d4e1a23e34f71858fc4a6dfbd4ccb => generated 0 bytes in 38 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 16:09:36.479311 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-768fa47d-4c5d-4743-b498-ce0a5535608f None None] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:09:36.480037 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-768fa47d-4c5d-4743-b498-ce0a5535608f None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:09:36.480523 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-768fa47d-4c5d-4743-b498-ce0a5535608f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:09:36.526937 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.auth.core [None req-768fa47d-4c5d-4743-b498-ce0a5535608f None None] MFA Rules not processed for user `e1eaf25bc6ce4ce1b8ac9da3a6322f8d`. Rule list: `[]` (Enabled: `True`). {{(pid=81438) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:09:36.528551 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-768fa47d-4c5d-4743-b498-ce0a5535608f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:09:36.626273 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1123/2281] 10.222.0.50 () {58 vars in 958 bytes} [Tue Sep 13 16:09:36 2022] POST /identity/v3/auth/tokens => generated 3329 bytes in 148 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:09:36.633946 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99c11cab-e462-45d5-b926-86ee9a8b742d admin admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:09:36.635744 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-99c11cab-e462-45d5-b926-86ee9a8b742d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:09:36.708706 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99c11cab-e462-45d5-b926-86ee9a8b742d admin admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:09:36.713798 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e85ace67-759e-4d38-b8bf-f4fe81ce3b6b service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7fd6b5afd67b48f9b0f0597ccd4cf4df', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:09:36.714430 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e85ace67-759e-4d38-b8bf-f4fe81ce3b6b service cinder] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:09:36.714713 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e85ace67-759e-4d38-b8bf-f4fe81ce3b6b service cinder] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:09:36.715023 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e85ace67-759e-4d38-b8bf-f4fe81ce3b6b service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:09:36.785553 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e85ace67-759e-4d38-b8bf-f4fe81ce3b6b service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:09:36.787923 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e85ace67-759e-4d38-b8bf-f4fe81ce3b6b service cinder] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:09:36.800475 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1159/2282] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 16:09:36 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 167 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 16:09:38.552389 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-768fa47d-4c5d-4743-b498-ce0a5535608f None None] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:09:38.554691 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-768fa47d-4c5d-4743-b498-ce0a5535608f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:09:38.575522 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-768fa47d-4c5d-4743-b498-ce0a5535608f None None] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:09:38.585940 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7f40202-41ab-4837-b8e2-b1de8edadeb4 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7fd6b5afd67b48f9b0f0597ccd4cf4df', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:09:38.586959 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7f40202-41ab-4837-b8e2-b1de8edadeb4 service cinder] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:09:38.589719 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7f40202-41ab-4837-b8e2-b1de8edadeb4 service cinder] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:09:38.590696 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7f40202-41ab-4837-b8e2-b1de8edadeb4 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:09:38.723697 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7f40202-41ab-4837-b8e2-b1de8edadeb4 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:09:38.725793 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7f40202-41ab-4837-b8e2-b1de8edadeb4 service cinder] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:09:38.739662 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1124/2283] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 16:09:38 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 188 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 16:09:40.507827 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e85ace67-759e-4d38-b8bf-f4fe81ce3b6b service cinder] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:09:40.516305 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e85ace67-759e-4d38-b8bf-f4fe81ce3b6b service cinder] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:09:40.520533 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39a5f03f-43a9-4022-af14-fbd2617f5b24 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7fd6b5afd67b48f9b0f0597ccd4cf4df', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:09:40.521140 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39a5f03f-43a9-4022-af14-fbd2617f5b24 service cinder] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:09:40.521446 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39a5f03f-43a9-4022-af14-fbd2617f5b24 service cinder] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:09:40.521734 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39a5f03f-43a9-4022-af14-fbd2617f5b24 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:09:40.532835 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39a5f03f-43a9-4022-af14-fbd2617f5b24 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:09:40.534404 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39a5f03f-43a9-4022-af14-fbd2617f5b24 service cinder] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:09:40.544956 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1160/2284] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 16:09:40 2022] GET /identity/v3/auth/tokens => generated 3329 bytes in 38 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 16:09:47.835494 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7f40202-41ab-4837-b8e2-b1de8edadeb4 service cinder] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:09:47.898377 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7f40202-41ab-4837-b8e2-b1de8edadeb4 service cinder] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:09:47.902976 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93ca4e8f-1e24-4f8a-a63f-3fa0dbd577af admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:09:47.903607 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93ca4e8f-1e24-4f8a-a63f-3fa0dbd577af admin admin] REQUEST_METHOD: `DELETE` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:09:47.903887 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93ca4e8f-1e24-4f8a-a63f-3fa0dbd577af admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:09:47.904162 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93ca4e8f-1e24-4f8a-a63f-3fa0dbd577af admin admin] PATH_INFO: `/v3/users/756e733b0cf14173aeb0f85b44bfbfaf` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:09:47.905279 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93ca4e8f-1e24-4f8a-a63f-3fa0dbd577af admin admin] RBAC: Authorizing `identity:delete_user(user_id=756e733b0cf14173aeb0f85b44bfbfaf)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:09:47.907236 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93ca4e8f-1e24-4f8a-a63f-3fa0dbd577af admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:09:47.921236 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=99,keystone:DELETE=1 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:09:47.953456 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-93ca4e8f-1e24-4f8a-a63f-3fa0dbd577af admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '756e733b0cf14173aeb0f85b44bfbfaf'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:09:47.960443 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-93ca4e8f-1e24-4f8a-a63f-3fa0dbd577af admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '756e733b0cf14173aeb0f85b44bfbfaf'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:09:47.973004 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-93ca4e8f-1e24-4f8a-a63f-3fa0dbd577af admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '756e733b0cf14173aeb0f85b44bfbfaf'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:09:47.973644 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-93ca4e8f-1e24-4f8a-a63f-3fa0dbd577af admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '756e733b0cf14173aeb0f85b44bfbfaf'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:09:47.980464 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-93ca4e8f-1e24-4f8a-a63f-3fa0dbd577af admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '756e733b0cf14173aeb0f85b44bfbfaf'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:09:47.987130 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1125/2285] 10.222.0.50 () {58 vars in 1224 bytes} [Tue Sep 13 16:09:47 2022] DELETE /identity/v3/users/756e733b0cf14173aeb0f85b44bfbfaf => generated 0 bytes in 152 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 16:09:47.991596 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-450a4f1b-3285-4927-8826-c16a05f5733a None None] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:09:47.992084 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-450a4f1b-3285-4927-8826-c16a05f5733a None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:09:47.992199 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-450a4f1b-3285-4927-8826-c16a05f5733a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:09:48.011522 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.auth.core [None req-450a4f1b-3285-4927-8826-c16a05f5733a None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81437) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:09:48.012722 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-450a4f1b-3285-4927-8826-c16a05f5733a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:09:48.072163 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1161/2286] 10.222.0.50 () {58 vars in 958 bytes} [Tue Sep 13 16:09:47 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 82 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:09:48.083935 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93ca4e8f-1e24-4f8a-a63f-3fa0dbd577af admin admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:09:48.085297 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-93ca4e8f-1e24-4f8a-a63f-3fa0dbd577af admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:09:48.145072 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93ca4e8f-1e24-4f8a-a63f-3fa0dbd577af admin admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:09:48.148789 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f1c53ae-d99c-490d-b05c-4cf38b0de643 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f436f07939d4855a21b195bc7566244', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:09:48.149396 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f1c53ae-d99c-490d-b05c-4cf38b0de643 service neutron] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:09:48.149680 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f1c53ae-d99c-490d-b05c-4cf38b0de643 service neutron] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:09:48.149958 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f1c53ae-d99c-490d-b05c-4cf38b0de643 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:09:48.159543 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f1c53ae-d99c-490d-b05c-4cf38b0de643 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:09:48.160961 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f1c53ae-d99c-490d-b05c-4cf38b0de643 service neutron] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:09:48.170261 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1126/2287] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 16:09:48 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 87 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 16:09:48.476668 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-450a4f1b-3285-4927-8826-c16a05f5733a None None] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:09:48.478123 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-450a4f1b-3285-4927-8826-c16a05f5733a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:09:48.489767 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-450a4f1b-3285-4927-8826-c16a05f5733a None None] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:09:48.494672 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5eef285c-f079-42d7-8aa0-77fc4a75e8ef admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:09:48.495342 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5eef285c-f079-42d7-8aa0-77fc4a75e8ef admin admin] REQUEST_METHOD: `DELETE` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:09:48.495625 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5eef285c-f079-42d7-8aa0-77fc4a75e8ef admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:09:48.495898 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5eef285c-f079-42d7-8aa0-77fc4a75e8ef admin admin] PATH_INFO: `/v3/projects/e9ad96d64d5c431098e52f0bd8c0473c` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:09:48.497166 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5eef285c-f079-42d7-8aa0-77fc4a75e8ef admin admin] RBAC: Authorizing `identity:delete_project(project_id=e9ad96d64d5c431098e52f0bd8c0473c)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:09:48.499011 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5eef285c-f079-42d7-8aa0-77fc4a75e8ef admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:09:48.551047 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=98,keystone:DELETE=2 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:09:48.555160 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-5eef285c-f079-42d7-8aa0-77fc4a75e8ef admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'e9ad96d64d5c431098e52f0bd8c0473c', 'request_id': 'req-5eef285c-f079-42d7-8aa0-77fc4a75e8ef'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:09:48.559076 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-5eef285c-f079-42d7-8aa0-77fc4a75e8ef admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'e9ad96d64d5c431098e52f0bd8c0473c', 'request_id': 'req-5eef285c-f079-42d7-8aa0-77fc4a75e8ef'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:09:48.566144 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-5eef285c-f079-42d7-8aa0-77fc4a75e8ef admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'e9ad96d64d5c431098e52f0bd8c0473c', 'request_id': 'req-5eef285c-f079-42d7-8aa0-77fc4a75e8ef'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:09:48.576614 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-5eef285c-f079-42d7-8aa0-77fc4a75e8ef admin admin] The token cache is being invalidate because project e9ad96d64d5c431098e52f0bd8c0473c was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81437) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 13 16:09:48.576899 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-5eef285c-f079-42d7-8aa0-77fc4a75e8ef admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:09:48.577546 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-5eef285c-f079-42d7-8aa0-77fc4a75e8ef admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:09:48.579047 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1162/2288] 10.222.0.50 () {58 vars in 1233 bytes} [Tue Sep 13 16:09:48 2022] DELETE /identity/v3/projects/e9ad96d64d5c431098e52f0bd8c0473c => generated 0 bytes in 103 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 16:09:48.583887 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f1c53ae-d99c-490d-b05c-4cf38b0de643 service neutron] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:09:48.593333 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f1c53ae-d99c-490d-b05c-4cf38b0de643 service neutron] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:09:48.653134 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29b330ff-1821-43f9-8546-1e6d57bb15f6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:09:48.653766 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29b330ff-1821-43f9-8546-1e6d57bb15f6 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:09:48.654043 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29b330ff-1821-43f9-8546-1e6d57bb15f6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:09:48.654314 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29b330ff-1821-43f9-8546-1e6d57bb15f6 admin admin] PATH_INFO: `/v3/users/1598727725b3437b9a8f36471293f677` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:09:48.655445 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29b330ff-1821-43f9-8546-1e6d57bb15f6 admin admin] RBAC: Authorizing `identity:delete_user(user_id=1598727725b3437b9a8f36471293f677)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:09:48.657332 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29b330ff-1821-43f9-8546-1e6d57bb15f6 admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:09:48.695424 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=85,keystone:DELETE=14,keystone:INSERT=1 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:09:48.706073 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-29b330ff-1821-43f9-8546-1e6d57bb15f6 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '1598727725b3437b9a8f36471293f677'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:09:48.712842 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-29b330ff-1821-43f9-8546-1e6d57bb15f6 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '1598727725b3437b9a8f36471293f677'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:09:48.724966 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-29b330ff-1821-43f9-8546-1e6d57bb15f6 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '1598727725b3437b9a8f36471293f677'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:09:48.725619 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-29b330ff-1821-43f9-8546-1e6d57bb15f6 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '1598727725b3437b9a8f36471293f677'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:09:48.732499 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-29b330ff-1821-43f9-8546-1e6d57bb15f6 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '1598727725b3437b9a8f36471293f677'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:09:48.739008 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1127/2289] 10.222.0.50 () {58 vars in 1224 bytes} [Tue Sep 13 16:09:48 2022] DELETE /identity/v3/users/1598727725b3437b9a8f36471293f677 => generated 0 bytes in 156 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 16:09:49.020058 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5eef285c-f079-42d7-8aa0-77fc4a75e8ef admin admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:09:49.021540 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-5eef285c-f079-42d7-8aa0-77fc4a75e8ef admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:09:49.034975 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5eef285c-f079-42d7-8aa0-77fc4a75e8ef admin admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:09:49.095454 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25c8e949-bbb8-47a1-8e06-0072ad11812f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:09:49.096774 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25c8e949-bbb8-47a1-8e06-0072ad11812f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:09:49.097078 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25c8e949-bbb8-47a1-8e06-0072ad11812f admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:09:49.097349 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25c8e949-bbb8-47a1-8e06-0072ad11812f admin admin] PATH_INFO: `/v3/projects/b3bfb0ffd8244b2c840847473ff3c993` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:09:49.098458 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25c8e949-bbb8-47a1-8e06-0072ad11812f admin admin] RBAC: Authorizing `identity:delete_project(project_id=b3bfb0ffd8244b2c840847473ff3c993)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:09:49.100446 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25c8e949-bbb8-47a1-8e06-0072ad11812f admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:09:49.146975 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-25c8e949-bbb8-47a1-8e06-0072ad11812f admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'b3bfb0ffd8244b2c840847473ff3c993', 'request_id': 'req-25c8e949-bbb8-47a1-8e06-0072ad11812f'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:09:49.153274 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-25c8e949-bbb8-47a1-8e06-0072ad11812f admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'b3bfb0ffd8244b2c840847473ff3c993', 'request_id': 'req-25c8e949-bbb8-47a1-8e06-0072ad11812f'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:09:49.160129 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-25c8e949-bbb8-47a1-8e06-0072ad11812f admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'b3bfb0ffd8244b2c840847473ff3c993', 'request_id': 'req-25c8e949-bbb8-47a1-8e06-0072ad11812f'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:09:49.170512 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-25c8e949-bbb8-47a1-8e06-0072ad11812f admin admin] The token cache is being invalidate because project b3bfb0ffd8244b2c840847473ff3c993 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81437) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 13 16:09:49.170800 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-25c8e949-bbb8-47a1-8e06-0072ad11812f admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:09:49.171456 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-25c8e949-bbb8-47a1-8e06-0072ad11812f admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:09:49.172942 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1163/2290] 10.222.0.50 () {58 vars in 1233 bytes} [Tue Sep 13 16:09:49 2022] DELETE /identity/v3/projects/b3bfb0ffd8244b2c840847473ff3c993 => generated 0 bytes in 153 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 16:09:49.177650 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29b330ff-1821-43f9-8546-1e6d57bb15f6 admin admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:09:49.179256 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-29b330ff-1821-43f9-8546-1e6d57bb15f6 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:09:49.191932 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29b330ff-1821-43f9-8546-1e6d57bb15f6 admin admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:09:49.248276 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86e82689-6448-4c77-9f5d-7f476a13bc9c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:09:49.248897 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86e82689-6448-4c77-9f5d-7f476a13bc9c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:09:49.249175 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86e82689-6448-4c77-9f5d-7f476a13bc9c admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:09:49.249447 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86e82689-6448-4c77-9f5d-7f476a13bc9c admin admin] PATH_INFO: `/v3/users/e1eaf25bc6ce4ce1b8ac9da3a6322f8d` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:09:49.250803 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86e82689-6448-4c77-9f5d-7f476a13bc9c admin admin] RBAC: Authorizing `identity:delete_user(user_id=e1eaf25bc6ce4ce1b8ac9da3a6322f8d)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:09:49.253089 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86e82689-6448-4c77-9f5d-7f476a13bc9c admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:09:49.298406 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-86e82689-6448-4c77-9f5d-7f476a13bc9c admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'e1eaf25bc6ce4ce1b8ac9da3a6322f8d'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:09:49.307058 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-86e82689-6448-4c77-9f5d-7f476a13bc9c admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'e1eaf25bc6ce4ce1b8ac9da3a6322f8d'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:09:49.318987 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-86e82689-6448-4c77-9f5d-7f476a13bc9c admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'e1eaf25bc6ce4ce1b8ac9da3a6322f8d'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:09:49.319633 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-86e82689-6448-4c77-9f5d-7f476a13bc9c admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'e1eaf25bc6ce4ce1b8ac9da3a6322f8d'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:09:49.326576 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-86e82689-6448-4c77-9f5d-7f476a13bc9c admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'e1eaf25bc6ce4ce1b8ac9da3a6322f8d'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:09:49.332487 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1128/2291] 10.222.0.50 () {58 vars in 1224 bytes} [Tue Sep 13 16:09:49 2022] DELETE /identity/v3/users/e1eaf25bc6ce4ce1b8ac9da3a6322f8d => generated 0 bytes in 155 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 16:09:49.598832 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25c8e949-bbb8-47a1-8e06-0072ad11812f admin admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:09:49.600327 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-25c8e949-bbb8-47a1-8e06-0072ad11812f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:09:49.613385 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25c8e949-bbb8-47a1-8e06-0072ad11812f admin admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:09:49.672320 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=86,keystone:DELETE=12,keystone:INSERT=2 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:09:49.691647 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f28763d0-4af0-458e-996a-4b41a879f49e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:09:49.692292 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f28763d0-4af0-458e-996a-4b41a879f49e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:09:49.692566 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f28763d0-4af0-458e-996a-4b41a879f49e admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:09:49.692837 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f28763d0-4af0-458e-996a-4b41a879f49e admin admin] PATH_INFO: `/v3/projects/387d492fa25d49ef867c67942c2a7c68` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:09:49.694286 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f28763d0-4af0-458e-996a-4b41a879f49e admin admin] RBAC: Authorizing `identity:delete_project(project_id=387d492fa25d49ef867c67942c2a7c68)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:09:49.696493 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f28763d0-4af0-458e-996a-4b41a879f49e admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:09:49.744148 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-f28763d0-4af0-458e-996a-4b41a879f49e admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '387d492fa25d49ef867c67942c2a7c68', 'request_id': 'req-f28763d0-4af0-458e-996a-4b41a879f49e'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:09:49.752488 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-f28763d0-4af0-458e-996a-4b41a879f49e admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '387d492fa25d49ef867c67942c2a7c68', 'request_id': 'req-f28763d0-4af0-458e-996a-4b41a879f49e'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:09:49.758798 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-f28763d0-4af0-458e-996a-4b41a879f49e admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '387d492fa25d49ef867c67942c2a7c68', 'request_id': 'req-f28763d0-4af0-458e-996a-4b41a879f49e'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:09:49.769586 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-f28763d0-4af0-458e-996a-4b41a879f49e admin admin] The token cache is being invalidate because project 387d492fa25d49ef867c67942c2a7c68 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81437) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 13 16:09:49.769876 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-f28763d0-4af0-458e-996a-4b41a879f49e admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:09:49.770510 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-f28763d0-4af0-458e-996a-4b41a879f49e admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:09:49.772025 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1164/2292] 10.222.0.50 () {58 vars in 1233 bytes} [Tue Sep 13 16:09:49 2022] DELETE /identity/v3/projects/387d492fa25d49ef867c67942c2a7c68 => generated 0 bytes in 174 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 16:09:52.696037 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1886ddbf-85eb-4ae9-a75f-2a7686e25698 None None] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:09:52.696541 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1886ddbf-85eb-4ae9-a75f-2a7686e25698 None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:09:52.696636 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1886ddbf-85eb-4ae9-a75f-2a7686e25698 None None] PATH_INFO: `/` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:09:52.697953 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1129/2293] 10.222.0.50 () {54 vars in 921 bytes} [Tue Sep 13 16:09:52 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 16:09:52.703800 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3aff565c-1f6d-416f-95ab-ceea8c3b0d59 None None] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:09:52.704258 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3aff565c-1f6d-416f-95ab-ceea8c3b0d59 None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:09:52.704470 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3aff565c-1f6d-416f-95ab-ceea8c3b0d59 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:09:52.724214 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.auth.core [None req-3aff565c-1f6d-416f-95ab-ceea8c3b0d59 None None] MFA Rules not processed for user `4e039b46c5e3445bbb39904b00c5430e`. Rule list: `[]` (Enabled: `True`). {{(pid=81437) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:09:52.725432 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-3aff565c-1f6d-416f-95ab-ceea8c3b0d59 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:09:52.805844 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1165/2294] 10.222.0.50 () {58 vars in 1018 bytes} [Tue Sep 13 16:09:52 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:09:52.818157 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1886ddbf-85eb-4ae9-a75f-2a7686e25698 None None] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:09:52.819952 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-1886ddbf-85eb-4ae9-a75f-2a7686e25698 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:09:52.858321 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=83,keystone:DELETE=15,keystone:INSERT=2 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:09:52.883914 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1886ddbf-85eb-4ae9-a75f-2a7686e25698 None None] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:09:52.887580 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5313887-af9b-4dd0-b841-9e6a8411f6f2 service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6ab5f8018f464be69d4af7683f553738', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:09:52.888191 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5313887-af9b-4dd0-b841-9e6a8411f6f2 service glance] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:09:52.888469 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5313887-af9b-4dd0-b841-9e6a8411f6f2 service glance] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:09:52.888742 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5313887-af9b-4dd0-b841-9e6a8411f6f2 service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:09:52.898837 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5313887-af9b-4dd0-b841-9e6a8411f6f2 service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:09:52.900298 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5313887-af9b-4dd0-b841-9e6a8411f6f2 service glance] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:09:52.914533 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1130/2295] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 16:09:52 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 97 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 16:09:52.989082 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-626d197d-ff29-4873-9bd8-2b145513b78e None None] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:09:52.989367 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-626d197d-ff29-4873-9bd8-2b145513b78e None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:09:52.989647 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-626d197d-ff29-4873-9bd8-2b145513b78e None None] PATH_INFO: `/` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:09:52.991016 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1166/2296] 10.222.0.50 () {54 vars in 921 bytes} [Tue Sep 13 16:09:52 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 16:09:52.995695 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e606da4f-953e-4eb3-aaf8-ac98d9385728 None None] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:09:52.995984 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e606da4f-953e-4eb3-aaf8-ac98d9385728 None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:09:52.996384 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e606da4f-953e-4eb3-aaf8-ac98d9385728 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:09:53.017962 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.auth.core [None req-e606da4f-953e-4eb3-aaf8-ac98d9385728 None None] MFA Rules not processed for user `2f436f07939d4855a21b195bc7566244`. Rule list: `[]` (Enabled: `True`). {{(pid=81438) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:09:53.019232 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-e606da4f-953e-4eb3-aaf8-ac98d9385728 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:09:53.072979 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1131/2297] 10.222.0.50 () {58 vars in 1018 bytes} [Tue Sep 13 16:09:52 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 78 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:09:53.073635 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-626d197d-ff29-4873-9bd8-2b145513b78e None None] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:09:53.075460 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-626d197d-ff29-4873-9bd8-2b145513b78e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:09:53.085025 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e606da4f-953e-4eb3-aaf8-ac98d9385728 None None] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:09:53.086554 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-e606da4f-953e-4eb3-aaf8-ac98d9385728 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:09:53.097851 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e606da4f-953e-4eb3-aaf8-ac98d9385728 None None] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:09:53.101866 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94428720-a00a-469b-9399-724b18fc68d7 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f436f07939d4855a21b195bc7566244', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:09:53.102473 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94428720-a00a-469b-9399-724b18fc68d7 service neutron] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:09:53.102782 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94428720-a00a-469b-9399-724b18fc68d7 service neutron] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:09:53.103086 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94428720-a00a-469b-9399-724b18fc68d7 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:09:53.110397 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94428720-a00a-469b-9399-724b18fc68d7 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:09:53.111846 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94428720-a00a-469b-9399-724b18fc68d7 service neutron] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:09:53.120461 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1132/2298] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 16:09:53 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 36 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 16:09:53.149897 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-626d197d-ff29-4873-9bd8-2b145513b78e None None] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:09:53.154492 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c444dec1-fa33-4571-97ec-1f4523232f60 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:09:53.155234 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c444dec1-fa33-4571-97ec-1f4523232f60 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:09:53.155512 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c444dec1-fa33-4571-97ec-1f4523232f60 admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:09:53.155784 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c444dec1-fa33-4571-97ec-1f4523232f60 admin admin] PATH_INFO: `/v3/users/e06b490d80174c7a974c94b4c8b38a99` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:09:53.156909 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c444dec1-fa33-4571-97ec-1f4523232f60 admin admin] RBAC: Authorizing `identity:delete_user(user_id=e06b490d80174c7a974c94b4c8b38a99)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:09:53.158852 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c444dec1-fa33-4571-97ec-1f4523232f60 admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:09:53.176066 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=91,keystone:DELETE=8,keystone:INSERT=1 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:09:53.207295 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5df05e00-2736-486c-b2d9-dc7abc3964a4 None None] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:09:53.207598 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5df05e00-2736-486c-b2d9-dc7abc3964a4 None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:09:53.207977 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5df05e00-2736-486c-b2d9-dc7abc3964a4 None None] PATH_INFO: `/` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:09:53.209491 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1133/2299] 10.222.0.50 () {54 vars in 921 bytes} [Tue Sep 13 16:09:53 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 16:09:53.213904 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-c444dec1-fa33-4571-97ec-1f4523232f60 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'e06b490d80174c7a974c94b4c8b38a99'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:09:53.215899 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5276556f-2671-4f53-8688-b595ce0cff00 None None] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:09:53.216149 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5276556f-2671-4f53-8688-b595ce0cff00 None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:09:53.216426 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5276556f-2671-4f53-8688-b595ce0cff00 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:09:53.221158 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-c444dec1-fa33-4571-97ec-1f4523232f60 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'e06b490d80174c7a974c94b4c8b38a99'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:09:53.235537 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-c444dec1-fa33-4571-97ec-1f4523232f60 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'e06b490d80174c7a974c94b4c8b38a99'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:09:53.236426 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-c444dec1-fa33-4571-97ec-1f4523232f60 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'e06b490d80174c7a974c94b4c8b38a99'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:09:53.237614 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.auth.core [None req-5276556f-2671-4f53-8688-b595ce0cff00 None None] MFA Rules not processed for user `4e039b46c5e3445bbb39904b00c5430e`. Rule list: `[]` (Enabled: `True`). {{(pid=81438) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:09:53.238834 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-5276556f-2671-4f53-8688-b595ce0cff00 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:09:53.244746 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-c444dec1-fa33-4571-97ec-1f4523232f60 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'e06b490d80174c7a974c94b4c8b38a99'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:09:53.248409 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1134/2300] 10.222.0.50 () {58 vars in 1018 bytes} [Tue Sep 13 16:09:53 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 13 16:09:53.251255 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1167/2301] 10.222.0.50 () {58 vars in 1224 bytes} [Tue Sep 13 16:09:53 2022] DELETE /identity/v3/users/e06b490d80174c7a974c94b4c8b38a99 => generated 0 bytes in 178 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 16:09:53.255746 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc7adbce-c26f-48d1-bb54-9ca8e3a55f57 None None] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:09:53.256032 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc7adbce-c26f-48d1-bb54-9ca8e3a55f57 None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:09:53.256309 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc7adbce-c26f-48d1-bb54-9ca8e3a55f57 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:09:53.259976 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c444dec1-fa33-4571-97ec-1f4523232f60 admin admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:09:53.261388 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-c444dec1-fa33-4571-97ec-1f4523232f60 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:09:53.277030 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.auth.core [None req-cc7adbce-c26f-48d1-bb54-9ca8e3a55f57 None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81438) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:09:53.278506 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-cc7adbce-c26f-48d1-bb54-9ca8e3a55f57 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:09:53.324711 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c444dec1-fa33-4571-97ec-1f4523232f60 admin admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:09:53.328429 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50cf046f-faf9-41db-8e1a-477a740561c1 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f436f07939d4855a21b195bc7566244', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:09:53.329031 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50cf046f-faf9-41db-8e1a-477a740561c1 service neutron] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:09:53.329308 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50cf046f-faf9-41db-8e1a-477a740561c1 service neutron] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:09:53.329574 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50cf046f-faf9-41db-8e1a-477a740561c1 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:09:53.343425 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1135/2302] 10.222.0.50 () {58 vars in 958 bytes} [Tue Sep 13 16:09:53 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 13 16:09:53.353751 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc7adbce-c26f-48d1-bb54-9ca8e3a55f57 None None] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:09:53.363606 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc7adbce-c26f-48d1-bb54-9ca8e3a55f57 None None] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:09:53.367243 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-818006d2-9df6-4e05-902a-8482c024560e service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f436f07939d4855a21b195bc7566244', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:09:53.367842 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-818006d2-9df6-4e05-902a-8482c024560e service neutron] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:09:53.368149 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-818006d2-9df6-4e05-902a-8482c024560e service neutron] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:09:53.368421 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-818006d2-9df6-4e05-902a-8482c024560e service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:09:53.377661 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-818006d2-9df6-4e05-902a-8482c024560e service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:09:53.379223 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-818006d2-9df6-4e05-902a-8482c024560e service neutron] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:09:53.388665 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1136/2303] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 16:09:53 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 35 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 13 16:09:53.399261 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50cf046f-faf9-41db-8e1a-477a740561c1 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:09:53.400690 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50cf046f-faf9-41db-8e1a-477a740561c1 service neutron] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:09:53.412784 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1168/2304] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 16:09:53 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 153 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 13 16:09:53.725914 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-818006d2-9df6-4e05-902a-8482c024560e service neutron] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:09:53.727617 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-818006d2-9df6-4e05-902a-8482c024560e service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:09:53.747302 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-818006d2-9df6-4e05-902a-8482c024560e service neutron] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:09:53.752827 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8409d48-e506-428d-a686-ef47b13fce8e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:09:53.753535 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8409d48-e506-428d-a686-ef47b13fce8e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:09:53.753816 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8409d48-e506-428d-a686-ef47b13fce8e admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:09:53.754242 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8409d48-e506-428d-a686-ef47b13fce8e admin admin] PATH_INFO: `/v3/projects/8c9e03206f954da5a29ab5d6e9881745` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:09:53.755434 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8409d48-e506-428d-a686-ef47b13fce8e admin admin] RBAC: Authorizing `identity:delete_project(project_id=8c9e03206f954da5a29ab5d6e9881745)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:09:53.757382 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8409d48-e506-428d-a686-ef47b13fce8e admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:09:53.808803 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=96,keystone:DELETE=4 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:09:53.810488 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-d8409d48-e506-428d-a686-ef47b13fce8e admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '8c9e03206f954da5a29ab5d6e9881745', 'request_id': 'req-d8409d48-e506-428d-a686-ef47b13fce8e'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:09:53.821020 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-d8409d48-e506-428d-a686-ef47b13fce8e admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '8c9e03206f954da5a29ab5d6e9881745', 'request_id': 'req-d8409d48-e506-428d-a686-ef47b13fce8e'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:09:53.827053 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-d8409d48-e506-428d-a686-ef47b13fce8e admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '8c9e03206f954da5a29ab5d6e9881745', 'request_id': 'req-d8409d48-e506-428d-a686-ef47b13fce8e'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:09:53.838835 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-d8409d48-e506-428d-a686-ef47b13fce8e admin admin] The token cache is being invalidate because project 8c9e03206f954da5a29ab5d6e9881745 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81438) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 13 16:09:53.839163 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-d8409d48-e506-428d-a686-ef47b13fce8e admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:09:53.839817 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-d8409d48-e506-428d-a686-ef47b13fce8e admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:09:53.841303 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1137/2305] 10.222.0.50 () {58 vars in 1233 bytes} [Tue Sep 13 16:09:53 2022] DELETE /identity/v3/projects/8c9e03206f954da5a29ab5d6e9881745 => generated 0 bytes in 116 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 16:09:56.582790 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7796d58f-bb12-4be7-bf0d-9c1b519577d3 None None] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:09:56.583303 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7796d58f-bb12-4be7-bf0d-9c1b519577d3 None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:09:56.583412 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7796d58f-bb12-4be7-bf0d-9c1b519577d3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:09:56.606642 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.auth.core [None req-7796d58f-bb12-4be7-bf0d-9c1b519577d3 None None] MFA Rules not processed for user `f8364b1a190d4218bf353a89bdb81448`. Rule list: `[]` (Enabled: `True`). {{(pid=81437) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:09:56.607962 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-7796d58f-bb12-4be7-bf0d-9c1b519577d3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:09:56.647124 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=90,keystone:DELETE=9,keystone:INSERT=1 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:09:56.676321 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1169/2306] 10.222.0.50 () {58 vars in 1009 bytes} [Tue Sep 13 16:09:56 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:09:56.688243 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8409d48-e506-428d-a686-ef47b13fce8e admin admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:09:56.689754 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-d8409d48-e506-428d-a686-ef47b13fce8e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:09:56.762274 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8409d48-e506-428d-a686-ef47b13fce8e admin admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:09:56.767241 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d8bc21c-5d55-4c82-966c-6038dae7a74d service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b00ab0d6321d483188bc69deaddcdbbc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:09:56.767852 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d8bc21c-5d55-4c82-966c-6038dae7a74d service swift] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:09:56.768148 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d8bc21c-5d55-4c82-966c-6038dae7a74d service swift] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:09:56.769134 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d8bc21c-5d55-4c82-966c-6038dae7a74d service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:09:56.784691 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d8bc21c-5d55-4c82-966c-6038dae7a74d service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:09:56.786352 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d8bc21c-5d55-4c82-966c-6038dae7a74d service swift] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:09:56.795740 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1138/2307] 10.222.0.50 () {60 vars in 1384 bytes} [Tue Sep 13 16:09:56 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 108 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Sep 13 16:10:06.670029 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=10 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:10:06.791873 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=39,keystone:INSERT=1,keystone:DELETE=3 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:10:10.802414 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7796d58f-bb12-4be7-bf0d-9c1b519577d3 None None] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:10:10.804462 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-7796d58f-bb12-4be7-bf0d-9c1b519577d3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:10:10.879677 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7796d58f-bb12-4be7-bf0d-9c1b519577d3 None None] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:10:10.885172 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b5f1f9d-db57-4a05-9244-625da9896986 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '31480b61418340bd88583f034f67412e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:10:10.885784 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b5f1f9d-db57-4a05-9244-625da9896986 service placement] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:10:10.886064 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b5f1f9d-db57-4a05-9244-625da9896986 service placement] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:10:10.886344 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b5f1f9d-db57-4a05-9244-625da9896986 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:10:10.888105 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-0b5f1f9d-db57-4a05-9244-625da9896986 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:10:10.901030 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b5f1f9d-db57-4a05-9244-625da9896986 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:10:10.903365 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b5f1f9d-db57-4a05-9244-625da9896986 service placement] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:10:10.912955 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1170/2308] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 16:10:10 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 111 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 16:10:19.946520 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35e1474c-47cf-4fce-887a-1ba509466300 None None] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:10:19.947185 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35e1474c-47cf-4fce-887a-1ba509466300 None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:10:19.947185 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35e1474c-47cf-4fce-887a-1ba509466300 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:10:19.967899 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.auth.core [None req-35e1474c-47cf-4fce-887a-1ba509466300 None None] MFA Rules not processed for user `313ac02befbf44ebabf3b41f84245c1f`. Rule list: `[]` (Enabled: `True`). {{(pid=81438) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:10:19.969032 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-35e1474c-47cf-4fce-887a-1ba509466300 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:10:20.033014 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1139/2309] 10.222.0.50 () {58 vars in 958 bytes} [Tue Sep 13 16:10:19 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:10:20.908460 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=32 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:10:21.655580 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74d0dc63-a380-45e0-8bcd-c370970cea47 None None] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:10:21.655863 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74d0dc63-a380-45e0-8bcd-c370970cea47 None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:10:21.656171 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74d0dc63-a380-45e0-8bcd-c370970cea47 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:10:21.676168 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.auth.core [None req-74d0dc63-a380-45e0-8bcd-c370970cea47 None None] MFA Rules not processed for user `77c7141135ae4b8fb3c4cbc3fa153687`. Rule list: `[]` (Enabled: `True`). {{(pid=81437) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:10:21.677533 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-74d0dc63-a380-45e0-8bcd-c370970cea47 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:10:21.734137 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1171/2310] 10.222.0.50 () {58 vars in 958 bytes} [Tue Sep 13 16:10:21 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 80 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:10:24.568889 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f8aa566-a6d7-46bb-9bb7-feae8b6112d8 None None] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:10:24.569686 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f8aa566-a6d7-46bb-9bb7-feae8b6112d8 None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:10:24.569991 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f8aa566-a6d7-46bb-9bb7-feae8b6112d8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:10:24.602628 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.auth.core [None req-5f8aa566-a6d7-46bb-9bb7-feae8b6112d8 None None] MFA Rules not processed for user `f8364b1a190d4218bf353a89bdb81448`. Rule list: `[]` (Enabled: `True`). {{(pid=81438) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:10:24.603794 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-5f8aa566-a6d7-46bb-9bb7-feae8b6112d8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:10:24.613102 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1140/2311] 10.222.0.50 () {58 vars in 967 bytes} [Tue Sep 13 16:10:24 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 45 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:10:24.628318 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74d0dc63-a380-45e0-8bcd-c370970cea47 None None] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:10:24.645179 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74d0dc63-a380-45e0-8bcd-c370970cea47 None None] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:10:24.651420 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36fe458a-c2f9-409a-8e31-2083e5d2cb3b service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b00ab0d6321d483188bc69deaddcdbbc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:10:24.652245 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36fe458a-c2f9-409a-8e31-2083e5d2cb3b service swift] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:10:24.652696 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36fe458a-c2f9-409a-8e31-2083e5d2cb3b service swift] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:10:24.653135 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36fe458a-c2f9-409a-8e31-2083e5d2cb3b service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:10:24.663350 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36fe458a-c2f9-409a-8e31-2083e5d2cb3b service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:10:24.664961 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36fe458a-c2f9-409a-8e31-2083e5d2cb3b service swift] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:10:24.678296 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1172/2312] 10.222.0.50 () {60 vars in 1384 bytes} [Tue Sep 13 16:10:24 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 50 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Sep 13 16:10:34.593126 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=28 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:10:34.673742 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=31 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:11:35.315030 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55c96f59-8c58-4b01-a430-235bb074fcc2 None None] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:11:35.315515 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55c96f59-8c58-4b01-a430-235bb074fcc2 None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:11:35.315619 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55c96f59-8c58-4b01-a430-235bb074fcc2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:11:35.337985 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.auth.core [None req-55c96f59-8c58-4b01-a430-235bb074fcc2 None None] MFA Rules not processed for user `6d31b2ec391441f7bab3546d2354b0a1`. Rule list: `[]` (Enabled: `True`). {{(pid=81438) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:11:35.339218 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-55c96f59-8c58-4b01-a430-235bb074fcc2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:11:35.449534 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1141/2313] 10.222.0.50 () {58 vars in 958 bytes} [Tue Sep 13 16:11:35 2022] POST /identity/v3/auth/tokens => generated 3268 bytes in 135 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 13 16:11:45.438675 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=25 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:12:07.031224 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36fe458a-c2f9-409a-8e31-2083e5d2cb3b service swift] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:12:07.032632 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-36fe458a-c2f9-409a-8e31-2083e5d2cb3b service swift] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:12:07.089239 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36fe458a-c2f9-409a-8e31-2083e5d2cb3b service swift] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:12:07.092877 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b5415e5-192a-4ebc-a513-5314cb072bc6 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7fd6b5afd67b48f9b0f0597ccd4cf4df', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:12:07.093717 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b5415e5-192a-4ebc-a513-5314cb072bc6 service cinder] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:12:07.093991 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b5415e5-192a-4ebc-a513-5314cb072bc6 service cinder] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:12:07.094259 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b5415e5-192a-4ebc-a513-5314cb072bc6 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:12:07.096029 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-9b5415e5-192a-4ebc-a513-5314cb072bc6 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:12:07.156158 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b5415e5-192a-4ebc-a513-5314cb072bc6 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:12:07.157620 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b5415e5-192a-4ebc-a513-5314cb072bc6 service cinder] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:12:07.166482 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1173/2314] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 16:12:07 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 136 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 16:12:07.979298 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55c96f59-8c58-4b01-a430-235bb074fcc2 None None] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:12:07.980704 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-55c96f59-8c58-4b01-a430-235bb074fcc2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:12:08.054459 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55c96f59-8c58-4b01-a430-235bb074fcc2 None None] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:12:08.059971 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9531dd8c-f17f-4f8b-89a0-28565c8eca2b service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4e039b46c5e3445bbb39904b00c5430e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'admin', 'ResellerAdmin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:12:08.060580 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9531dd8c-f17f-4f8b-89a0-28565c8eca2b service nova] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:12:08.060860 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9531dd8c-f17f-4f8b-89a0-28565c8eca2b service nova] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:12:08.061133 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9531dd8c-f17f-4f8b-89a0-28565c8eca2b service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:12:08.063623 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-9531dd8c-f17f-4f8b-89a0-28565c8eca2b service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:12:08.120804 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9531dd8c-f17f-4f8b-89a0-28565c8eca2b service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:12:08.122478 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9531dd8c-f17f-4f8b-89a0-28565c8eca2b service nova] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:12:08.132914 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1142/2315] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 16:12:07 2022] GET /identity/v3/auth/tokens => generated 3266 bytes in 154 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 16:12:12.537864 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b5415e5-192a-4ebc-a513-5314cb072bc6 service cinder] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:12:12.547175 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b5415e5-192a-4ebc-a513-5314cb072bc6 service cinder] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:12:12.550728 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1051937b-5c3e-465c-aa58-0af2daac5704 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7fd6b5afd67b48f9b0f0597ccd4cf4df', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:12:12.551357 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1051937b-5c3e-465c-aa58-0af2daac5704 service cinder] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:12:12.551633 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1051937b-5c3e-465c-aa58-0af2daac5704 service cinder] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:12:12.551908 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1051937b-5c3e-465c-aa58-0af2daac5704 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:12:12.553634 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-1051937b-5c3e-465c-aa58-0af2daac5704 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:12:12.565083 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1051937b-5c3e-465c-aa58-0af2daac5704 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:12:12.566490 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1051937b-5c3e-465c-aa58-0af2daac5704 service cinder] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:12:12.575165 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1174/2316] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 16:12:12 2022] GET /identity/v3/auth/tokens => generated 3268 bytes in 38 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 16:12:12.682043 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9531dd8c-f17f-4f8b-89a0-28565c8eca2b service nova] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:12:12.692114 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9531dd8c-f17f-4f8b-89a0-28565c8eca2b service nova] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:12:12.695666 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-741875fc-1893-4b36-a3cb-4e448b4c5317 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7fd6b5afd67b48f9b0f0597ccd4cf4df', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:12:12.696266 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-741875fc-1893-4b36-a3cb-4e448b4c5317 service cinder] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:12:12.696547 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-741875fc-1893-4b36-a3cb-4e448b4c5317 service cinder] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:12:12.696825 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-741875fc-1893-4b36-a3cb-4e448b4c5317 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:12:12.698531 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-741875fc-1893-4b36-a3cb-4e448b4c5317 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:12:12.711491 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-741875fc-1893-4b36-a3cb-4e448b4c5317 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:12:12.712892 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-741875fc-1893-4b36-a3cb-4e448b4c5317 service cinder] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:12:12.723592 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1143/2317] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 16:12:12 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 42 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 16:12:22.571288 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=54 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:12:22.718839 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=62 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:13:05.411673 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1051937b-5c3e-465c-aa58-0af2daac5704 service cinder] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:13:05.413122 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-1051937b-5c3e-465c-aa58-0af2daac5704 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:13:05.477594 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1051937b-5c3e-465c-aa58-0af2daac5704 service cinder] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:13:05.481464 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1efca8f1-2159-4e8c-a567-79d6d5b735d2 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f436f07939d4855a21b195bc7566244', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:13:05.482071 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1efca8f1-2159-4e8c-a567-79d6d5b735d2 service neutron] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:13:05.482357 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1efca8f1-2159-4e8c-a567-79d6d5b735d2 service neutron] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:13:05.482631 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1efca8f1-2159-4e8c-a567-79d6d5b735d2 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:13:05.484504 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-1efca8f1-2159-4e8c-a567-79d6d5b735d2 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:13:05.499766 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1efca8f1-2159-4e8c-a567-79d6d5b735d2 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:13:05.501191 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1efca8f1-2159-4e8c-a567-79d6d5b735d2 service neutron] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:13:05.509006 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1175/2318] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 16:13:05 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 98 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 16:13:15.505501 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=26 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:13:44.145091 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-741875fc-1893-4b36-a3cb-4e448b4c5317 service cinder] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:13:44.146673 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-741875fc-1893-4b36-a3cb-4e448b4c5317 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:13:44.160428 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-741875fc-1893-4b36-a3cb-4e448b4c5317 service cinder] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:13:44.166650 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d30293dd-84fe-42ae-96e1-d66ac4889fe0 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4e039b46c5e3445bbb39904b00c5430e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'admin', 'ResellerAdmin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:13:44.167324 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d30293dd-84fe-42ae-96e1-d66ac4889fe0 service nova] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:13:44.167606 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d30293dd-84fe-42ae-96e1-d66ac4889fe0 service nova] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:13:44.167916 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d30293dd-84fe-42ae-96e1-d66ac4889fe0 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:13:44.169638 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-d30293dd-84fe-42ae-96e1-d66ac4889fe0 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:13:44.181164 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d30293dd-84fe-42ae-96e1-d66ac4889fe0 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:13:44.182756 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d30293dd-84fe-42ae-96e1-d66ac4889fe0 service nova] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:13:44.191787 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1144/2319] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 16:13:44 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 47 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 16:13:49.960115 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1efca8f1-2159-4e8c-a567-79d6d5b735d2 service neutron] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:13:49.961532 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-1efca8f1-2159-4e8c-a567-79d6d5b735d2 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:13:49.973305 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1efca8f1-2159-4e8c-a567-79d6d5b735d2 service neutron] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:13:49.976913 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ad3e24f-ba03-4031-a808-2a17a3f493d3 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7fd6b5afd67b48f9b0f0597ccd4cf4df', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:13:49.977504 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ad3e24f-ba03-4031-a808-2a17a3f493d3 service cinder] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:13:49.977783 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ad3e24f-ba03-4031-a808-2a17a3f493d3 service cinder] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:13:49.978056 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ad3e24f-ba03-4031-a808-2a17a3f493d3 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:13:49.979735 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-5ad3e24f-ba03-4031-a808-2a17a3f493d3 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:13:49.990392 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ad3e24f-ba03-4031-a808-2a17a3f493d3 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:13:49.991845 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ad3e24f-ba03-4031-a808-2a17a3f493d3 service cinder] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:13:50.001311 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1176/2320] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 16:13:49 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 42 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 16:13:54.187583 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=6 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:13:59.996616 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=6 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:15:11.693483 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cca459d-425b-4aba-ab7c-32ea6eb22164 None None] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:15:11.693999 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cca459d-425b-4aba-ab7c-32ea6eb22164 None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:15:11.693999 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cca459d-425b-4aba-ab7c-32ea6eb22164 None None] PATH_INFO: `/` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:15:11.695364 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1145/2321] 10.222.0.50 () {54 vars in 961 bytes} [Tue Sep 13 16:15:11 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 16:15:11.700387 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36dcaff3-b65a-44ae-9df7-40700e80cb30 None None] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:15:11.700838 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36dcaff3-b65a-44ae-9df7-40700e80cb30 None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:15:11.700939 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36dcaff3-b65a-44ae-9df7-40700e80cb30 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:15:11.907018 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.auth.core [None req-36dcaff3-b65a-44ae-9df7-40700e80cb30 None None] MFA Rules not processed for user `31480b61418340bd88583f034f67412e`. Rule list: `[]` (Enabled: `True`). {{(pid=81437) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:15:11.908157 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-36dcaff3-b65a-44ae-9df7-40700e80cb30 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:15:11.965865 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1177/2322] 10.222.0.50 () {58 vars in 1058 bytes} [Tue Sep 13 16:15:11 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 266 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:15:11.973285 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cca459d-425b-4aba-ab7c-32ea6eb22164 None None] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:15:11.987313 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cca459d-425b-4aba-ab7c-32ea6eb22164 None None] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:15:11.992327 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a587a003-d63b-4a6d-a949-f57d19ed389d service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '31480b61418340bd88583f034f67412e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:15:11.992916 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a587a003-d63b-4a6d-a949-f57d19ed389d service placement] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:15:11.993192 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a587a003-d63b-4a6d-a949-f57d19ed389d service placement] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:15:11.993462 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a587a003-d63b-4a6d-a949-f57d19ed389d service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:15:12.002165 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a587a003-d63b-4a6d-a949-f57d19ed389d service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:15:12.003763 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a587a003-d63b-4a6d-a949-f57d19ed389d service placement] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:15:12.014088 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1146/2323] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 16:15:11 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 41 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 16:15:21.964130 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=48 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:15:22.009798 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=6 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:17:07.927437 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36dcaff3-b65a-44ae-9df7-40700e80cb30 None None] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:17:07.956849 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36dcaff3-b65a-44ae-9df7-40700e80cb30 None None] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:17:07.962306 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31df8369-04f9-4198-b836-843316c69f19 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4e039b46c5e3445bbb39904b00c5430e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'admin', 'ResellerAdmin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:17:07.962918 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31df8369-04f9-4198-b836-843316c69f19 service nova] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:17:07.963224 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31df8369-04f9-4198-b836-843316c69f19 service nova] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:17:07.963532 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31df8369-04f9-4198-b836-843316c69f19 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:17:07.972521 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31df8369-04f9-4198-b836-843316c69f19 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:17:07.974091 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31df8369-04f9-4198-b836-843316c69f19 service nova] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:17:08.001454 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1178/2324] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 16:17:07 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 75 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 16:17:08.828750 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a587a003-d63b-4a6d-a949-f57d19ed389d service placement] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:17:08.848301 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a587a003-d63b-4a6d-a949-f57d19ed389d service placement] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:17:08.851845 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9584f411-1264-4306-a9fa-c4293d645eb0 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7fd6b5afd67b48f9b0f0597ccd4cf4df', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:17:08.852432 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9584f411-1264-4306-a9fa-c4293d645eb0 service cinder] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:17:08.852710 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9584f411-1264-4306-a9fa-c4293d645eb0 service cinder] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:17:08.852979 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9584f411-1264-4306-a9fa-c4293d645eb0 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:17:08.861326 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9584f411-1264-4306-a9fa-c4293d645eb0 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:17:08.862751 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9584f411-1264-4306-a9fa-c4293d645eb0 service cinder] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:17:08.892865 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1147/2325] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 16:17:08 2022] GET /identity/v3/auth/tokens => generated 3266 bytes in 65 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 16:17:13.151558 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31df8369-04f9-4198-b836-843316c69f19 service nova] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:17:13.162726 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31df8369-04f9-4198-b836-843316c69f19 service nova] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:17:13.169553 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-263f3b4d-e2a1-4ab1-af7b-aff364f2825f service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4e039b46c5e3445bbb39904b00c5430e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'admin', 'ResellerAdmin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:17:13.169914 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-263f3b4d-e2a1-4ab1-af7b-aff364f2825f service nova] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:17:13.170194 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-263f3b4d-e2a1-4ab1-af7b-aff364f2825f service nova] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:17:13.170464 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-263f3b4d-e2a1-4ab1-af7b-aff364f2825f service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:17:13.178675 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-263f3b4d-e2a1-4ab1-af7b-aff364f2825f service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:17:13.180403 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-263f3b4d-e2a1-4ab1-af7b-aff364f2825f service nova] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:17:13.208534 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1179/2326] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 16:17:13 2022] GET /identity/v3/auth/tokens => generated 3268 bytes in 57 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 16:17:18.889962 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=17 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:17:23.206046 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=33 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:18:12.412453 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9584f411-1264-4306-a9fa-c4293d645eb0 service cinder] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:18:12.432044 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9584f411-1264-4306-a9fa-c4293d645eb0 service cinder] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:18:12.435791 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e31c968-80d2-4d1f-8232-dbb85b7f0003 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f436f07939d4855a21b195bc7566244', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:18:12.436398 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e31c968-80d2-4d1f-8232-dbb85b7f0003 service neutron] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:18:12.436681 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e31c968-80d2-4d1f-8232-dbb85b7f0003 service neutron] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:18:12.436961 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e31c968-80d2-4d1f-8232-dbb85b7f0003 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:18:12.444396 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e31c968-80d2-4d1f-8232-dbb85b7f0003 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:18:12.445813 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e31c968-80d2-4d1f-8232-dbb85b7f0003 service neutron] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:18:12.454808 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1148/2327] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 16:18:12 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 42 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 16:18:22.450764 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=10 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:18:44.156290 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-263f3b4d-e2a1-4ab1-af7b-aff364f2825f service nova] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:18:44.165598 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-263f3b4d-e2a1-4ab1-af7b-aff364f2825f service nova] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:18:44.169111 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec949367-24ed-40fd-9231-c14645861158 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7fd6b5afd67b48f9b0f0597ccd4cf4df', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:18:44.169695 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec949367-24ed-40fd-9231-c14645861158 service cinder] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:18:44.169966 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec949367-24ed-40fd-9231-c14645861158 service cinder] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:18:44.170231 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec949367-24ed-40fd-9231-c14645861158 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:18:44.178606 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec949367-24ed-40fd-9231-c14645861158 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:18:44.179997 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec949367-24ed-40fd-9231-c14645861158 service cinder] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:18:44.206985 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1180/2328] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 16:18:44 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 52 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 16:18:49.693708 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e31c968-80d2-4d1f-8232-dbb85b7f0003 service neutron] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:18:49.702798 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e31c968-80d2-4d1f-8232-dbb85b7f0003 service neutron] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:18:49.706591 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9737927-5ffe-4783-97f9-2c092071ba48 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7fd6b5afd67b48f9b0f0597ccd4cf4df', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:18:49.707233 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9737927-5ffe-4783-97f9-2c092071ba48 service cinder] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:18:49.707514 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9737927-5ffe-4783-97f9-2c092071ba48 service cinder] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:18:49.707793 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9737927-5ffe-4783-97f9-2c092071ba48 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:18:49.716023 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9737927-5ffe-4783-97f9-2c092071ba48 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:18:49.717440 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9737927-5ffe-4783-97f9-2c092071ba48 service cinder] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:18:49.745335 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1149/2329] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 16:18:49 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 52 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 16:18:54.204545 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=13 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:18:59.742724 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=13 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:20:16.742744 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec949367-24ed-40fd-9231-c14645861158 service cinder] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:20:16.744409 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-ec949367-24ed-40fd-9231-c14645861158 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:20:16.812190 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec949367-24ed-40fd-9231-c14645861158 service cinder] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:20:16.817152 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58d6821d-7fb1-4593-a2d0-82ac4b013275 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '31480b61418340bd88583f034f67412e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:20:16.817747 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58d6821d-7fb1-4593-a2d0-82ac4b013275 service placement] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:20:16.818022 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58d6821d-7fb1-4593-a2d0-82ac4b013275 service placement] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:20:16.818292 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58d6821d-7fb1-4593-a2d0-82ac4b013275 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:20:16.820122 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-58d6821d-7fb1-4593-a2d0-82ac4b013275 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:20:16.831355 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58d6821d-7fb1-4593-a2d0-82ac4b013275 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:20:16.832925 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58d6821d-7fb1-4593-a2d0-82ac4b013275 service placement] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:20:16.843105 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1181/2330] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 16:20:16 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 101 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 16:20:26.838904 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=32 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:22:08.463794 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9737927-5ffe-4783-97f9-2c092071ba48 service cinder] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:22:08.465435 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-f9737927-5ffe-4783-97f9-2c092071ba48 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:22:08.548508 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9737927-5ffe-4783-97f9-2c092071ba48 service cinder] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:22:08.554246 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2f62848-a6a6-4559-bb3d-f7b126f7b514 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4e039b46c5e3445bbb39904b00c5430e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'admin', 'ResellerAdmin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:22:08.554847 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2f62848-a6a6-4559-bb3d-f7b126f7b514 service nova] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:22:08.555166 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2f62848-a6a6-4559-bb3d-f7b126f7b514 service nova] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:22:08.555442 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2f62848-a6a6-4559-bb3d-f7b126f7b514 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:22:08.557238 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-d2f62848-a6a6-4559-bb3d-f7b126f7b514 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:22:08.626108 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2f62848-a6a6-4559-bb3d-f7b126f7b514 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:22:08.627634 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2f62848-a6a6-4559-bb3d-f7b126f7b514 service nova] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:22:08.637490 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1150/2331] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 16:22:08 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 174 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 16:22:08.700028 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58d6821d-7fb1-4593-a2d0-82ac4b013275 service placement] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:22:08.754193 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58d6821d-7fb1-4593-a2d0-82ac4b013275 service placement] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:22:08.757794 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea47ce07-3f03-43b7-a6e2-e95e4f9923f0 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7fd6b5afd67b48f9b0f0597ccd4cf4df', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:22:08.758392 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea47ce07-3f03-43b7-a6e2-e95e4f9923f0 service cinder] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:22:08.758670 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea47ce07-3f03-43b7-a6e2-e95e4f9923f0 service cinder] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:22:08.758982 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea47ce07-3f03-43b7-a6e2-e95e4f9923f0 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:22:08.760852 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-ea47ce07-3f03-43b7-a6e2-e95e4f9923f0 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:22:08.829700 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea47ce07-3f03-43b7-a6e2-e95e4f9923f0 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:22:08.831170 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea47ce07-3f03-43b7-a6e2-e95e4f9923f0 service cinder] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:22:08.840075 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1182/2332] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 16:22:08 2022] GET /identity/v3/auth/tokens => generated 3266 bytes in 141 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 16:22:13.635040 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2f62848-a6a6-4559-bb3d-f7b126f7b514 service nova] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:22:13.644292 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2f62848-a6a6-4559-bb3d-f7b126f7b514 service nova] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:22:13.648381 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f816e11a-08af-429b-82e7-fc3856ca23b3 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7fd6b5afd67b48f9b0f0597ccd4cf4df', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:22:13.648971 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f816e11a-08af-429b-82e7-fc3856ca23b3 service cinder] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:22:13.649244 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f816e11a-08af-429b-82e7-fc3856ca23b3 service cinder] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:22:13.649516 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f816e11a-08af-429b-82e7-fc3856ca23b3 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:22:13.651325 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-f816e11a-08af-429b-82e7-fc3856ca23b3 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:22:13.721802 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f816e11a-08af-429b-82e7-fc3856ca23b3 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:22:13.723959 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f816e11a-08af-429b-82e7-fc3856ca23b3 service cinder] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:22:13.733004 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1151/2333] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 16:22:13 2022] GET /identity/v3/auth/tokens => generated 3268 bytes in 98 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 16:22:18.836003 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=53 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:22:23.728716 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=99 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:22:44.024252 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea47ce07-3f03-43b7-a6e2-e95e4f9923f0 service cinder] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:22:44.034304 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea47ce07-3f03-43b7-a6e2-e95e4f9923f0 service cinder] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:22:44.037900 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a3cb7ef-4890-4c75-8252-39d66f689293 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f436f07939d4855a21b195bc7566244', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:22:44.038484 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a3cb7ef-4890-4c75-8252-39d66f689293 service neutron] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:22:44.038757 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a3cb7ef-4890-4c75-8252-39d66f689293 service neutron] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:22:44.039067 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a3cb7ef-4890-4c75-8252-39d66f689293 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:22:44.041563 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-5a3cb7ef-4890-4c75-8252-39d66f689293 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:22:44.055027 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a3cb7ef-4890-4c75-8252-39d66f689293 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:22:44.056735 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a3cb7ef-4890-4c75-8252-39d66f689293 service neutron] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:22:44.067644 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1183/2334] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 16:22:44 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 44 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 16:22:45.247257 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f816e11a-08af-429b-82e7-fc3856ca23b3 service cinder] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:22:45.256389 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f816e11a-08af-429b-82e7-fc3856ca23b3 service cinder] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:22:45.260308 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bad498ff-a3e4-40f9-8ffd-3c280f89163b service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f436f07939d4855a21b195bc7566244', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:22:45.260907 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bad498ff-a3e4-40f9-8ffd-3c280f89163b service neutron] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:22:45.261188 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bad498ff-a3e4-40f9-8ffd-3c280f89163b service neutron] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:22:45.261522 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bad498ff-a3e4-40f9-8ffd-3c280f89163b service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:22:45.263644 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-bad498ff-a3e4-40f9-8ffd-3c280f89163b service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:22:45.276986 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bad498ff-a3e4-40f9-8ffd-3c280f89163b service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:22:45.278396 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bad498ff-a3e4-40f9-8ffd-3c280f89163b service neutron] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:22:45.289302 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1152/2335] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 16:22:45 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 43 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 16:22:54.062743 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=6 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:22:55.284508 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=6 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:23:18.452450 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a3cb7ef-4890-4c75-8252-39d66f689293 service neutron] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:23:18.454210 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-5a3cb7ef-4890-4c75-8252-39d66f689293 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:23:18.524351 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a3cb7ef-4890-4c75-8252-39d66f689293 service neutron] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:23:18.528055 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-baba755c-b3ec-4d06-85b1-a644a0839993 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f436f07939d4855a21b195bc7566244', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:23:18.528649 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-baba755c-b3ec-4d06-85b1-a644a0839993 service neutron] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:23:18.528928 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-baba755c-b3ec-4d06-85b1-a644a0839993 service neutron] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:23:18.529199 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-baba755c-b3ec-4d06-85b1-a644a0839993 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:23:18.531635 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-baba755c-b3ec-4d06-85b1-a644a0839993 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:23:18.540702 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-baba755c-b3ec-4d06-85b1-a644a0839993 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:23:18.542112 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-baba755c-b3ec-4d06-85b1-a644a0839993 service neutron] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:23:18.550350 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1184/2336] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 16:23:18 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 99 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 16:23:28.546921 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=31 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:23:44.616795 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bad498ff-a3e4-40f9-8ffd-3c280f89163b service neutron] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:23:44.618398 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-bad498ff-a3e4-40f9-8ffd-3c280f89163b service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:23:44.632642 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bad498ff-a3e4-40f9-8ffd-3c280f89163b service neutron] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:23:44.638076 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5dc06781-4798-4637-a784-c521c73e0b9f service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4e039b46c5e3445bbb39904b00c5430e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'admin', 'ResellerAdmin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:23:44.638665 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dc06781-4798-4637-a784-c521c73e0b9f service nova] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:23:44.638974 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dc06781-4798-4637-a784-c521c73e0b9f service nova] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:23:44.639251 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dc06781-4798-4637-a784-c521c73e0b9f service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:23:44.640999 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-5dc06781-4798-4637-a784-c521c73e0b9f service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:23:44.711470 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5dc06781-4798-4637-a784-c521c73e0b9f service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:23:44.713089 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5dc06781-4798-4637-a784-c521c73e0b9f service nova] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:23:44.723095 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1153/2337] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 16:23:44 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 107 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 16:23:49.897944 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-baba755c-b3ec-4d06-85b1-a644a0839993 service neutron] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:23:49.911170 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-baba755c-b3ec-4d06-85b1-a644a0839993 service neutron] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:23:49.916776 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3363ffe-9bfe-4daf-b35a-09a617fe1d51 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4e039b46c5e3445bbb39904b00c5430e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'admin', 'ResellerAdmin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:23:49.917366 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3363ffe-9bfe-4daf-b35a-09a617fe1d51 service nova] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:23:49.917640 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3363ffe-9bfe-4daf-b35a-09a617fe1d51 service nova] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:23:49.917908 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3363ffe-9bfe-4daf-b35a-09a617fe1d51 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:23:49.973878 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3363ffe-9bfe-4daf-b35a-09a617fe1d51 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:23:49.975517 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3363ffe-9bfe-4daf-b35a-09a617fe1d51 service nova] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:23:49.984288 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1185/2338] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 16:23:49 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 87 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 16:23:54.718752 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=33 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:23:59.980192 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=28 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:25:24.753530 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5dc06781-4798-4637-a784-c521c73e0b9f service nova] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:25:24.755151 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-5dc06781-4798-4637-a784-c521c73e0b9f service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:25:24.843428 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5dc06781-4798-4637-a784-c521c73e0b9f service nova] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:25:24.848495 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce0c628a-7007-499f-a65e-3637e0561fd9 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '31480b61418340bd88583f034f67412e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:25:24.849132 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce0c628a-7007-499f-a65e-3637e0561fd9 service placement] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:25:24.849413 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce0c628a-7007-499f-a65e-3637e0561fd9 service placement] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:25:24.849687 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce0c628a-7007-499f-a65e-3637e0561fd9 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:25:24.858645 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce0c628a-7007-499f-a65e-3637e0561fd9 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:25:24.860282 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce0c628a-7007-499f-a65e-3637e0561fd9 service placement] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:25:24.871329 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1154/2339] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 16:25:24 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 118 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 16:25:34.865814 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=37 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:27:08.314455 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3363ffe-9bfe-4daf-b35a-09a617fe1d51 service nova] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:27:08.338334 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce0c628a-7007-499f-a65e-3637e0561fd9 service placement] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:27:08.339938 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-ce0c628a-7007-499f-a65e-3637e0561fd9 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:27:08.366498 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce0c628a-7007-499f-a65e-3637e0561fd9 service placement] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:27:08.374513 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eba17fa4-384f-4686-91e4-ca4dab65c440 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7fd6b5afd67b48f9b0f0597ccd4cf4df', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:27:08.375453 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eba17fa4-384f-4686-91e4-ca4dab65c440 service cinder] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:27:08.375736 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eba17fa4-384f-4686-91e4-ca4dab65c440 service cinder] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:27:08.376417 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eba17fa4-384f-4686-91e4-ca4dab65c440 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:27:08.382762 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3363ffe-9bfe-4daf-b35a-09a617fe1d51 service nova] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:27:08.385489 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eba17fa4-384f-4686-91e4-ca4dab65c440 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:27:08.387501 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eba17fa4-384f-4686-91e4-ca4dab65c440 service cinder] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:27:08.389333 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fd01de1-59cc-42c6-a71a-5fc769a18315 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4e039b46c5e3445bbb39904b00c5430e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'admin', 'ResellerAdmin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:27:08.389939 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fd01de1-59cc-42c6-a71a-5fc769a18315 service nova] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:27:08.390215 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fd01de1-59cc-42c6-a71a-5fc769a18315 service nova] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:27:08.390564 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fd01de1-59cc-42c6-a71a-5fc769a18315 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:27:08.400219 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4fd01de1-59cc-42c6-a71a-5fc769a18315 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:27:08.402011 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4fd01de1-59cc-42c6-a71a-5fc769a18315 service nova] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:27:08.412225 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1186/2340] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 16:27:08 2022] GET /identity/v3/auth/tokens => generated 3266 bytes in 98 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 16:27:08.417933 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1155/2341] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 16:27:08 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 80 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 13 16:27:12.860140 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fd01de1-59cc-42c6-a71a-5fc769a18315 service nova] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:27:12.869236 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fd01de1-59cc-42c6-a71a-5fc769a18315 service nova] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:27:12.872847 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b5697ee-8836-45b6-8e1d-9a980dd6adeb service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f436f07939d4855a21b195bc7566244', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:27:12.873431 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b5697ee-8836-45b6-8e1d-9a980dd6adeb service neutron] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:27:12.873701 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b5697ee-8836-45b6-8e1d-9a980dd6adeb service neutron] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:27:12.873965 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b5697ee-8836-45b6-8e1d-9a980dd6adeb service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:27:12.882789 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b5697ee-8836-45b6-8e1d-9a980dd6adeb service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:27:12.884246 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b5697ee-8836-45b6-8e1d-9a980dd6adeb service neutron] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:27:12.894041 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1187/2342] 10.222.0.50 () {60 vars in 1396 bytes} [Tue Sep 13 16:27:12 2022] GET /identity/v3/auth/tokens?allow_expired=1 => generated 3268 bytes in 34 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 13 16:27:18.415158 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=18 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:27:22.888950 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=33 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:27:42.940707 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eba17fa4-384f-4686-91e4-ca4dab65c440 service cinder] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:27:42.950104 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eba17fa4-384f-4686-91e4-ca4dab65c440 service cinder] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:27:42.953731 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4a208c8-80c6-4b21-8a90-c7bae2b41b18 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f436f07939d4855a21b195bc7566244', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:27:42.954389 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4a208c8-80c6-4b21-8a90-c7bae2b41b18 service neutron] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:27:42.955219 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4a208c8-80c6-4b21-8a90-c7bae2b41b18 service neutron] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:27:42.955499 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4a208c8-80c6-4b21-8a90-c7bae2b41b18 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:27:42.966069 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4a208c8-80c6-4b21-8a90-c7bae2b41b18 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:27:42.967532 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4a208c8-80c6-4b21-8a90-c7bae2b41b18 service neutron] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:27:42.979368 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1156/2343] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 16:27:42 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 40 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 16:27:43.958765 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b5697ee-8836-45b6-8e1d-9a980dd6adeb service neutron] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:27:43.967280 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b5697ee-8836-45b6-8e1d-9a980dd6adeb service neutron] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:27:43.971843 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cafd1bc-c729-4d0d-a0d3-e1b3bae377ca service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f436f07939d4855a21b195bc7566244', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:27:43.972501 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cafd1bc-c729-4d0d-a0d3-e1b3bae377ca service neutron] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:27:43.972779 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cafd1bc-c729-4d0d-a0d3-e1b3bae377ca service neutron] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:27:43.973053 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cafd1bc-c729-4d0d-a0d3-e1b3bae377ca service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:27:43.983903 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5cafd1bc-c729-4d0d-a0d3-e1b3bae377ca service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:27:43.985418 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5cafd1bc-c729-4d0d-a0d3-e1b3bae377ca service neutron] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:27:43.997444 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1188/2344] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 16:27:43 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 39 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 16:27:52.974316 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=6 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:27:53.992474 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=6 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:28:21.485308 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4a208c8-80c6-4b21-8a90-c7bae2b41b18 service neutron] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:28:21.505330 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4a208c8-80c6-4b21-8a90-c7bae2b41b18 service neutron] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:28:21.509635 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a364695-92a5-48f2-9a44-e6c042fbd6ab service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f436f07939d4855a21b195bc7566244', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:28:21.510256 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a364695-92a5-48f2-9a44-e6c042fbd6ab service neutron] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:28:21.510517 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a364695-92a5-48f2-9a44-e6c042fbd6ab service neutron] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:28:21.510789 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a364695-92a5-48f2-9a44-e6c042fbd6ab service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:28:21.518695 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a364695-92a5-48f2-9a44-e6c042fbd6ab service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:28:21.520513 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a364695-92a5-48f2-9a44-e6c042fbd6ab service neutron] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:28:21.539243 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1157/2345] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 16:28:21 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 54 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 16:28:31.533895 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=10 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:28:38.821346 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cafd1bc-c729-4d0d-a0d3-e1b3bae377ca service neutron] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:28:38.836365 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cafd1bc-c729-4d0d-a0d3-e1b3bae377ca service neutron] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:28:38.842478 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d894ff40-2290-48b6-9448-3224ee641f44 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4e039b46c5e3445bbb39904b00c5430e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'admin', 'ResellerAdmin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:28:38.843146 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d894ff40-2290-48b6-9448-3224ee641f44 service nova] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:28:38.843425 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d894ff40-2290-48b6-9448-3224ee641f44 service nova] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:28:38.843696 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d894ff40-2290-48b6-9448-3224ee641f44 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:28:38.845556 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-d894ff40-2290-48b6-9448-3224ee641f44 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:28:38.859645 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d894ff40-2290-48b6-9448-3224ee641f44 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:28:38.861842 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d894ff40-2290-48b6-9448-3224ee641f44 service nova] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:28:38.883197 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1189/2346] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 16:28:38 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 63 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 16:28:44.857008 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a364695-92a5-48f2-9a44-e6c042fbd6ab service neutron] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:28:44.870570 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a364695-92a5-48f2-9a44-e6c042fbd6ab service neutron] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:28:44.876114 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-247ae2e1-bfe2-401f-a95f-214b37621401 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4e039b46c5e3445bbb39904b00c5430e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'admin', 'ResellerAdmin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:28:44.876790 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-247ae2e1-bfe2-401f-a95f-214b37621401 service nova] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:28:44.877040 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-247ae2e1-bfe2-401f-a95f-214b37621401 service nova] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:28:44.877267 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-247ae2e1-bfe2-401f-a95f-214b37621401 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:28:44.886101 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-247ae2e1-bfe2-401f-a95f-214b37621401 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:28:44.887712 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-247ae2e1-bfe2-401f-a95f-214b37621401 service nova] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:28:44.921133 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1158/2347] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 16:28:44 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 64 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 16:28:48.870129 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=6 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:28:54.912019 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=13 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:29:10.820811 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0799f710-1fb8-420e-998a-08040f254b29 None None] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:29:10.821239 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0799f710-1fb8-420e-998a-08040f254b29 None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:29:10.821345 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0799f710-1fb8-420e-998a-08040f254b29 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:29:10.863151 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.auth.core [None req-0799f710-1fb8-420e-998a-08040f254b29 None None] MFA Rules not processed for user `c0fd4bbe430b4a8888f3be4b23982c5a`. Rule list: `[]` (Enabled: `True`). {{(pid=81437) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:29:10.864709 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-0799f710-1fb8-420e-998a-08040f254b29 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:29:10.882447 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1190/2348] 10.222.0.50 () {58 vars in 958 bytes} [Tue Sep 13 16:29:10 2022] POST /identity/v3/auth/tokens => generated 3266 bytes in 63 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:29:11.213592 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-247ae2e1-bfe2-401f-a95f-214b37621401 service nova] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:29:11.222873 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-247ae2e1-bfe2-401f-a95f-214b37621401 service nova] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:29:11.227196 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdd26e57-2911-4b9f-9287-529307b5bd86 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f436f07939d4855a21b195bc7566244', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:29:11.227795 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdd26e57-2911-4b9f-9287-529307b5bd86 service neutron] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:29:11.228107 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdd26e57-2911-4b9f-9287-529307b5bd86 service neutron] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:29:11.228338 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdd26e57-2911-4b9f-9287-529307b5bd86 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:29:11.230045 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-fdd26e57-2911-4b9f-9287-529307b5bd86 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:29:11.239871 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdd26e57-2911-4b9f-9287-529307b5bd86 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:29:11.241425 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdd26e57-2911-4b9f-9287-529307b5bd86 service neutron] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:29:11.250029 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1159/2349] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 16:29:11 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 37 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 16:29:11.641596 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0799f710-1fb8-420e-998a-08040f254b29 None None] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:29:11.654085 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0799f710-1fb8-420e-998a-08040f254b29 None None] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:29:11.659332 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1aef9f6-8ffb-4464-b4d3-52acc3fbfb1f service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '31480b61418340bd88583f034f67412e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:29:11.660022 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1aef9f6-8ffb-4464-b4d3-52acc3fbfb1f service placement] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:29:11.660373 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1aef9f6-8ffb-4464-b4d3-52acc3fbfb1f service placement] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:29:11.660712 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1aef9f6-8ffb-4464-b4d3-52acc3fbfb1f service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:29:11.662666 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-d1aef9f6-8ffb-4464-b4d3-52acc3fbfb1f service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:29:11.674389 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1aef9f6-8ffb-4464-b4d3-52acc3fbfb1f service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:29:11.676062 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1aef9f6-8ffb-4464-b4d3-52acc3fbfb1f service placement] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:29:11.687620 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1191/2350] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 16:29:11 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 46 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 16:29:19.610072 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdd26e57-2911-4b9f-9287-529307b5bd86 service neutron] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:29:19.624895 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdd26e57-2911-4b9f-9287-529307b5bd86 service neutron] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:29:19.630288 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebecc679-319c-45a2-9d21-7126685a1e1e service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4e039b46c5e3445bbb39904b00c5430e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'admin', 'ResellerAdmin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:29:19.631558 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebecc679-319c-45a2-9d21-7126685a1e1e service nova] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:29:19.631928 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebecc679-319c-45a2-9d21-7126685a1e1e service nova] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:29:19.632262 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebecc679-319c-45a2-9d21-7126685a1e1e service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:29:19.633984 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-ebecc679-319c-45a2-9d21-7126685a1e1e service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:29:19.645937 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebecc679-319c-45a2-9d21-7126685a1e1e service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:29:19.648470 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebecc679-319c-45a2-9d21-7126685a1e1e service nova] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:29:19.658913 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1160/2351] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 16:29:19 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 49 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 16:29:21.682743 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=20 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:29:29.654361 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=12 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:29:33.219123 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1aef9f6-8ffb-4464-b4d3-52acc3fbfb1f service placement] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:29:33.245414 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1aef9f6-8ffb-4464-b4d3-52acc3fbfb1f service placement] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:29:33.249838 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d23a43b-aa22-4053-bb2c-72acd23425c8 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7fd6b5afd67b48f9b0f0597ccd4cf4df', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:29:33.250744 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d23a43b-aa22-4053-bb2c-72acd23425c8 service cinder] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:29:33.250744 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d23a43b-aa22-4053-bb2c-72acd23425c8 service cinder] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:29:33.250950 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d23a43b-aa22-4053-bb2c-72acd23425c8 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:29:33.252893 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-5d23a43b-aa22-4053-bb2c-72acd23425c8 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:29:33.289420 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d23a43b-aa22-4053-bb2c-72acd23425c8 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:29:33.290853 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d23a43b-aa22-4053-bb2c-72acd23425c8 service cinder] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:29:33.309896 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1192/2352] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 16:29:33 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 91 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 16:29:41.007210 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e2bf62a-166a-4176-8604-dcde44a422e3 None None] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:29:41.007743 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e2bf62a-166a-4176-8604-dcde44a422e3 None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:29:41.007743 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e2bf62a-166a-4176-8604-dcde44a422e3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:29:41.037697 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.auth.core [None req-5e2bf62a-166a-4176-8604-dcde44a422e3 None None] MFA Rules not processed for user `77c7141135ae4b8fb3c4cbc3fa153687`. Rule list: `[]` (Enabled: `True`). {{(pid=81438) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:29:41.039092 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-5e2bf62a-166a-4176-8604-dcde44a422e3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:29:41.050648 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1161/2353] 10.222.0.50 () {58 vars in 958 bytes} [Tue Sep 13 16:29:41 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 44 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:29:43.307413 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=23 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:29:51.034176 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=6 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:30:30.795065 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d23a43b-aa22-4053-bb2c-72acd23425c8 service cinder] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:30:30.796664 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-5d23a43b-aa22-4053-bb2c-72acd23425c8 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:30:30.864994 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d23a43b-aa22-4053-bb2c-72acd23425c8 service cinder] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:30:30.872428 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-135ab3cc-05c9-49ff-97c7-20f394c2b928 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '31480b61418340bd88583f034f67412e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:30:30.873066 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-135ab3cc-05c9-49ff-97c7-20f394c2b928 service placement] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:30:30.873339 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-135ab3cc-05c9-49ff-97c7-20f394c2b928 service placement] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:30:30.873608 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-135ab3cc-05c9-49ff-97c7-20f394c2b928 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:30:30.875468 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-135ab3cc-05c9-49ff-97c7-20f394c2b928 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:30:30.887048 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-135ab3cc-05c9-49ff-97c7-20f394c2b928 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:30:30.888640 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-135ab3cc-05c9-49ff-97c7-20f394c2b928 service placement] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:30:30.898808 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1193/2354] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 16:30:30 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 104 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 16:30:40.894191 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=32 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:32:08.681390 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e2bf62a-166a-4176-8604-dcde44a422e3 None None] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:32:08.683755 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-5e2bf62a-166a-4176-8604-dcde44a422e3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:32:08.769377 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e2bf62a-166a-4176-8604-dcde44a422e3 None None] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:32:08.774802 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44ef2b5b-2f4b-453a-8ff7-ced5b948fff6 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4e039b46c5e3445bbb39904b00c5430e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'admin', 'ResellerAdmin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:32:08.775432 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44ef2b5b-2f4b-453a-8ff7-ced5b948fff6 service nova] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:32:08.775766 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44ef2b5b-2f4b-453a-8ff7-ced5b948fff6 service nova] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:32:08.776019 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44ef2b5b-2f4b-453a-8ff7-ced5b948fff6 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:32:08.777803 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-44ef2b5b-2f4b-453a-8ff7-ced5b948fff6 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:32:08.846462 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44ef2b5b-2f4b-453a-8ff7-ced5b948fff6 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:32:08.848058 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44ef2b5b-2f4b-453a-8ff7-ced5b948fff6 service nova] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:32:08.857417 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1162/2355] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 16:32:08 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 177 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 16:32:18.852659 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=66 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:32:39.540081 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-135ab3cc-05c9-49ff-97c7-20f394c2b928 service placement] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:32:39.542445 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-135ab3cc-05c9-49ff-97c7-20f394c2b928 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:32:39.597914 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-135ab3cc-05c9-49ff-97c7-20f394c2b928 service placement] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:32:39.601442 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7cdb1af-af80-4550-8a3a-907fcfb049cd service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7fd6b5afd67b48f9b0f0597ccd4cf4df', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:32:39.602026 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7cdb1af-af80-4550-8a3a-907fcfb049cd service cinder] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:32:39.602298 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7cdb1af-af80-4550-8a3a-907fcfb049cd service cinder] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:32:39.602561 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7cdb1af-af80-4550-8a3a-907fcfb049cd service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:32:39.605342 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-a7cdb1af-af80-4550-8a3a-907fcfb049cd service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:32:39.675239 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7cdb1af-af80-4550-8a3a-907fcfb049cd service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:32:39.677209 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7cdb1af-af80-4550-8a3a-907fcfb049cd service cinder] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:32:39.687596 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1194/2356] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 16:32:39 2022] GET /identity/v3/auth/tokens => generated 3266 bytes in 148 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 16:32:39.731001 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44ef2b5b-2f4b-453a-8ff7-ced5b948fff6 service nova] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:32:39.740186 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44ef2b5b-2f4b-453a-8ff7-ced5b948fff6 service nova] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:32:39.744206 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f269a86-3b2b-442f-90c0-0d045767a1e0 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7fd6b5afd67b48f9b0f0597ccd4cf4df', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:32:39.744799 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f269a86-3b2b-442f-90c0-0d045767a1e0 service cinder] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:32:39.745076 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f269a86-3b2b-442f-90c0-0d045767a1e0 service cinder] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:32:39.745346 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f269a86-3b2b-442f-90c0-0d045767a1e0 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:32:39.747164 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-7f269a86-3b2b-442f-90c0-0d045767a1e0 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:32:39.835415 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f269a86-3b2b-442f-90c0-0d045767a1e0 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:32:39.836838 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f269a86-3b2b-442f-90c0-0d045767a1e0 service cinder] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:32:39.867776 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1163/2357] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 16:32:39 2022] GET /identity/v3/auth/tokens => generated 3268 bytes in 137 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 16:32:43.152692 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7cdb1af-af80-4550-8a3a-907fcfb049cd service cinder] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:32:43.161674 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7cdb1af-af80-4550-8a3a-907fcfb049cd service cinder] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:32:43.165316 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7967c24-035f-4c01-a628-d638f0c4aacc service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f436f07939d4855a21b195bc7566244', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:32:43.165905 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7967c24-035f-4c01-a628-d638f0c4aacc service neutron] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:32:43.166176 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7967c24-035f-4c01-a628-d638f0c4aacc service neutron] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:32:43.166441 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7967c24-035f-4c01-a628-d638f0c4aacc service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:32:43.176775 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7967c24-035f-4c01-a628-d638f0c4aacc service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:32:43.178186 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7967c24-035f-4c01-a628-d638f0c4aacc service neutron] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:32:43.188962 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1195/2358] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 16:32:43 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 37 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 16:32:44.296438 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f269a86-3b2b-442f-90c0-0d045767a1e0 service cinder] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:32:44.305542 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f269a86-3b2b-442f-90c0-0d045767a1e0 service cinder] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:32:44.309371 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a7bce53-b0cc-4ad2-b372-1eefbb63379b service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f436f07939d4855a21b195bc7566244', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:32:44.309969 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a7bce53-b0cc-4ad2-b372-1eefbb63379b service neutron] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:32:44.310285 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a7bce53-b0cc-4ad2-b372-1eefbb63379b service neutron] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:32:44.310557 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a7bce53-b0cc-4ad2-b372-1eefbb63379b service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:32:44.320576 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a7bce53-b0cc-4ad2-b372-1eefbb63379b service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:32:44.322101 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a7bce53-b0cc-4ad2-b372-1eefbb63379b service neutron] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:32:44.333299 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1164/2359] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 16:32:44 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 37 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 16:32:53.184284 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=59 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:32:54.328239 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=47 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:33:26.448199 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7967c24-035f-4c01-a628-d638f0c4aacc service neutron] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:33:26.449803 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-a7967c24-035f-4c01-a628-d638f0c4aacc service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:33:26.520570 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7967c24-035f-4c01-a628-d638f0c4aacc service neutron] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:33:26.524309 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09456955-b9c0-4ebb-9b48-366995aed23d service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f436f07939d4855a21b195bc7566244', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:33:26.524919 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09456955-b9c0-4ebb-9b48-366995aed23d service neutron] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:33:26.525195 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09456955-b9c0-4ebb-9b48-366995aed23d service neutron] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:33:26.525463 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09456955-b9c0-4ebb-9b48-366995aed23d service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:33:26.527337 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-09456955-b9c0-4ebb-9b48-366995aed23d service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:33:26.537443 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09456955-b9c0-4ebb-9b48-366995aed23d service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:33:26.538875 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09456955-b9c0-4ebb-9b48-366995aed23d service neutron] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:33:26.547556 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1196/2360] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 16:33:26 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 100 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 16:33:36.543269 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=31 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:34:38.297569 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b426f5aa-25bb-435f-8eab-c21e963f4bf3 None None] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:34:38.298092 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b426f5aa-25bb-435f-8eab-c21e963f4bf3 None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:34:38.298209 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b426f5aa-25bb-435f-8eab-c21e963f4bf3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:34:38.325988 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.auth.core [None req-b426f5aa-25bb-435f-8eab-c21e963f4bf3 None None] MFA Rules not processed for user `e1cbafa1d3c244c5906e475494b39d4c`. Rule list: `[]` (Enabled: `True`). {{(pid=81438) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:34:38.327392 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-b426f5aa-25bb-435f-8eab-c21e963f4bf3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:34:38.334129 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1165/2361] 10.222.0.50 () {58 vars in 958 bytes} [Tue Sep 13 16:34:38 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 38 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 13 16:34:38.638391 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09456955-b9c0-4ebb-9b48-366995aed23d service neutron] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:34:38.647118 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09456955-b9c0-4ebb-9b48-366995aed23d service neutron] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:34:38.651007 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3da785c9-9967-4166-9fc2-ebac5d7904ab service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f436f07939d4855a21b195bc7566244', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:34:38.651574 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3da785c9-9967-4166-9fc2-ebac5d7904ab service neutron] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:34:38.651845 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3da785c9-9967-4166-9fc2-ebac5d7904ab service neutron] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:34:38.652129 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3da785c9-9967-4166-9fc2-ebac5d7904ab service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:34:38.653798 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-3da785c9-9967-4166-9fc2-ebac5d7904ab service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:34:38.663248 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3da785c9-9967-4166-9fc2-ebac5d7904ab service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:34:38.664663 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3da785c9-9967-4166-9fc2-ebac5d7904ab service neutron] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:34:38.673149 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1197/2362] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 16:34:38 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 35 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 16:34:39.231349 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b426f5aa-25bb-435f-8eab-c21e963f4bf3 None None] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:34:39.244329 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b426f5aa-25bb-435f-8eab-c21e963f4bf3 None None] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:34:39.249672 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d977a33-23b7-45d7-9e49-cbf7ca70db95 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '31480b61418340bd88583f034f67412e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:34:39.250282 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d977a33-23b7-45d7-9e49-cbf7ca70db95 service placement] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:34:39.250566 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d977a33-23b7-45d7-9e49-cbf7ca70db95 service placement] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:34:39.250844 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d977a33-23b7-45d7-9e49-cbf7ca70db95 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:34:39.259755 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d977a33-23b7-45d7-9e49-cbf7ca70db95 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:34:39.261325 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d977a33-23b7-45d7-9e49-cbf7ca70db95 service placement] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:34:39.273966 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1166/2363] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 16:34:39 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 43 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 16:34:40.013174 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3da785c9-9967-4166-9fc2-ebac5d7904ab service neutron] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:34:40.022307 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3da785c9-9967-4166-9fc2-ebac5d7904ab service neutron] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:34:40.025828 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67e5f559-3467-41e4-8dae-b322939f3d07 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7fd6b5afd67b48f9b0f0597ccd4cf4df', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:34:40.026422 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67e5f559-3467-41e4-8dae-b322939f3d07 service cinder] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:34:40.026698 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67e5f559-3467-41e4-8dae-b322939f3d07 service cinder] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:34:40.027005 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67e5f559-3467-41e4-8dae-b322939f3d07 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:34:40.028794 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-67e5f559-3467-41e4-8dae-b322939f3d07 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:34:40.105665 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67e5f559-3467-41e4-8dae-b322939f3d07 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:34:40.107907 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67e5f559-3467-41e4-8dae-b322939f3d07 service cinder] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:34:40.117183 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1198/2364] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 16:34:40 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 105 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 16:34:41.879359 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d977a33-23b7-45d7-9e49-cbf7ca70db95 service placement] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:34:41.895442 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d977a33-23b7-45d7-9e49-cbf7ca70db95 service placement] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:34:41.902259 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5aae237-fb44-46c2-b386-2b572796ba47 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4e039b46c5e3445bbb39904b00c5430e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'admin', 'ResellerAdmin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:34:41.903021 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5aae237-fb44-46c2-b386-2b572796ba47 service nova] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:34:41.903374 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5aae237-fb44-46c2-b386-2b572796ba47 service nova] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:34:41.903662 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5aae237-fb44-46c2-b386-2b572796ba47 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:34:41.913396 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5aae237-fb44-46c2-b386-2b572796ba47 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:34:41.915014 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5aae237-fb44-46c2-b386-2b572796ba47 service nova] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:34:41.926034 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1167/2365] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 16:34:41 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 48 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 16:34:47.070321 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67e5f559-3467-41e4-8dae-b322939f3d07 service cinder] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:34:47.071910 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-67e5f559-3467-41e4-8dae-b322939f3d07 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:34:47.088860 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67e5f559-3467-41e4-8dae-b322939f3d07 service cinder] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:34:47.094594 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddd898c8-5087-4394-888b-f14ae84de0d0 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4e039b46c5e3445bbb39904b00c5430e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'admin', 'ResellerAdmin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:34:47.095233 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddd898c8-5087-4394-888b-f14ae84de0d0 service nova] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:34:47.095509 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddd898c8-5087-4394-888b-f14ae84de0d0 service nova] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:34:47.095784 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddd898c8-5087-4394-888b-f14ae84de0d0 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:34:47.104977 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ddd898c8-5087-4394-888b-f14ae84de0d0 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:34:47.106550 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ddd898c8-5087-4394-888b-f14ae84de0d0 service nova] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:34:47.117153 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1199/2366] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 16:34:47 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 47 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 13 16:34:51.921403 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=18 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:34:57.112815 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=45 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:35:34.592140 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5aae237-fb44-46c2-b386-2b572796ba47 service nova] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:35:34.602098 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5aae237-fb44-46c2-b386-2b572796ba47 service nova] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:35:34.605701 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2152940-1038-44aa-aba2-fa607935858e service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7fd6b5afd67b48f9b0f0597ccd4cf4df', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:35:34.606286 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2152940-1038-44aa-aba2-fa607935858e service cinder] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:35:34.606562 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2152940-1038-44aa-aba2-fa607935858e service cinder] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:35:34.606830 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2152940-1038-44aa-aba2-fa607935858e service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:35:34.663742 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2152940-1038-44aa-aba2-fa607935858e service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:35:34.665230 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2152940-1038-44aa-aba2-fa607935858e service cinder] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:35:34.675176 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1168/2367] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 16:35:34 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 83 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 16:35:34.730072 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddd898c8-5087-4394-888b-f14ae84de0d0 service nova] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:35:34.769173 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddd898c8-5087-4394-888b-f14ae84de0d0 service nova] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:35:34.774496 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68057608-879b-4981-9a48-491e877d8580 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '31480b61418340bd88583f034f67412e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:35:34.774859 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68057608-879b-4981-9a48-491e877d8580 service placement] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:35:34.775164 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68057608-879b-4981-9a48-491e877d8580 service placement] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:35:34.775444 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68057608-879b-4981-9a48-491e877d8580 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:35:34.784696 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68057608-879b-4981-9a48-491e877d8580 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:35:34.786263 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68057608-879b-4981-9a48-491e877d8580 service placement] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:35:34.796414 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1200/2368] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 16:35:34 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 67 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 16:35:44.670310 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=28 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:35:44.792508 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=13 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:37:35.059809 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2152940-1038-44aa-aba2-fa607935858e service cinder] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:37:35.061388 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-e2152940-1038-44aa-aba2-fa607935858e service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:37:35.104181 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2152940-1038-44aa-aba2-fa607935858e service cinder] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:37:35.107820 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-650ade13-2fdb-4dcd-a409-f1a396b53681 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7fd6b5afd67b48f9b0f0597ccd4cf4df', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:37:35.108422 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-650ade13-2fdb-4dcd-a409-f1a396b53681 service cinder] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:37:35.108705 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-650ade13-2fdb-4dcd-a409-f1a396b53681 service cinder] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:37:35.108979 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-650ade13-2fdb-4dcd-a409-f1a396b53681 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:37:35.130838 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-650ade13-2fdb-4dcd-a409-f1a396b53681 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:37:35.132285 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-650ade13-2fdb-4dcd-a409-f1a396b53681 service cinder] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:37:35.160964 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1169/2369] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 16:37:35 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 102 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 16:37:37.245293 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68057608-879b-4981-9a48-491e877d8580 service placement] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:37:37.333410 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68057608-879b-4981-9a48-491e877d8580 service placement] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:37:37.340707 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e326e32c-e814-48c1-8874-61c1173c8bd0 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4e039b46c5e3445bbb39904b00c5430e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'admin', 'ResellerAdmin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:37:37.341481 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e326e32c-e814-48c1-8874-61c1173c8bd0 service nova] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:37:37.341871 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e326e32c-e814-48c1-8874-61c1173c8bd0 service nova] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:37:37.342261 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e326e32c-e814-48c1-8874-61c1173c8bd0 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:37:37.345078 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-e326e32c-e814-48c1-8874-61c1173c8bd0 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:37:37.367562 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e326e32c-e814-48c1-8874-61c1173c8bd0 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:37:37.369176 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e326e32c-e814-48c1-8874-61c1173c8bd0 service nova] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:37:37.394155 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1201/2370] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 16:37:37 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 149 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 16:37:39.343658 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-650ade13-2fdb-4dcd-a409-f1a396b53681 service cinder] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:37:39.346794 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-650ade13-2fdb-4dcd-a409-f1a396b53681 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:37:39.359113 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-650ade13-2fdb-4dcd-a409-f1a396b53681 service cinder] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:37:39.364470 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db99f94a-aca0-4302-ad57-b84e8dd16d81 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f436f07939d4855a21b195bc7566244', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:37:39.365507 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db99f94a-aca0-4302-ad57-b84e8dd16d81 service neutron] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:37:39.369531 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db99f94a-aca0-4302-ad57-b84e8dd16d81 service neutron] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:37:39.370046 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db99f94a-aca0-4302-ad57-b84e8dd16d81 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:37:39.383579 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db99f94a-aca0-4302-ad57-b84e8dd16d81 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:37:39.386086 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db99f94a-aca0-4302-ad57-b84e8dd16d81 service neutron] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:37:39.396389 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1170/2371] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 16:37:39 2022] GET /identity/v3/auth/tokens => generated 3266 bytes in 53 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 13 16:37:41.037127 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb1e8fcb-fc82-4567-88c1-c752c0d38fd9 None None] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:37:41.037623 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb1e8fcb-fc82-4567-88c1-c752c0d38fd9 None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:37:41.037758 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb1e8fcb-fc82-4567-88c1-c752c0d38fd9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:37:41.057887 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.auth.core [None req-eb1e8fcb-fc82-4567-88c1-c752c0d38fd9 None None] MFA Rules not processed for user `77c7141135ae4b8fb3c4cbc3fa153687`. Rule list: `[]` (Enabled: `True`). {{(pid=81437) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:37:41.059124 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-eb1e8fcb-fc82-4567-88c1-c752c0d38fd9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:37:41.066866 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1202/2372] 10.222.0.50 () {58 vars in 958 bytes} [Tue Sep 13 16:37:41 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:37:41.374876 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db99f94a-aca0-4302-ad57-b84e8dd16d81 service neutron] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:37:41.383523 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db99f94a-aca0-4302-ad57-b84e8dd16d81 service neutron] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:37:41.387139 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89b4ab0d-8b51-44a9-b311-7c1afd9cb2bc service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f436f07939d4855a21b195bc7566244', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:37:41.387722 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89b4ab0d-8b51-44a9-b311-7c1afd9cb2bc service neutron] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:37:41.387996 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89b4ab0d-8b51-44a9-b311-7c1afd9cb2bc service neutron] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:37:41.388263 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89b4ab0d-8b51-44a9-b311-7c1afd9cb2bc service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:37:41.395878 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89b4ab0d-8b51-44a9-b311-7c1afd9cb2bc service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:37:41.397290 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89b4ab0d-8b51-44a9-b311-7c1afd9cb2bc service neutron] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:37:41.404951 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1171/2373] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 16:37:41 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 31 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 16:37:45.112757 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb1e8fcb-fc82-4567-88c1-c752c0d38fd9 None None] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:37:45.122298 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb1e8fcb-fc82-4567-88c1-c752c0d38fd9 None None] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:37:45.126975 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a183207e-e8b8-41d4-831d-1065991f2dfd service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f436f07939d4855a21b195bc7566244', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:37:45.127575 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a183207e-e8b8-41d4-831d-1065991f2dfd service neutron] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:37:45.127854 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a183207e-e8b8-41d4-831d-1065991f2dfd service neutron] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:37:45.128126 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a183207e-e8b8-41d4-831d-1065991f2dfd service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:37:45.129925 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-a183207e-e8b8-41d4-831d-1065991f2dfd service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:37:45.143271 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a183207e-e8b8-41d4-831d-1065991f2dfd service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:37:45.144682 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a183207e-e8b8-41d4-831d-1065991f2dfd service neutron] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:37:45.155198 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1203/2374] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 16:37:45 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 43 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 16:37:46.331070 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89b4ab0d-8b51-44a9-b311-7c1afd9cb2bc service neutron] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:37:46.339799 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89b4ab0d-8b51-44a9-b311-7c1afd9cb2bc service neutron] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:37:46.343577 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-474fec02-246f-4470-9217-3eec0a4460f7 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f436f07939d4855a21b195bc7566244', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:37:46.344165 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-474fec02-246f-4470-9217-3eec0a4460f7 service neutron] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:37:46.344444 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-474fec02-246f-4470-9217-3eec0a4460f7 service neutron] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:37:46.344716 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-474fec02-246f-4470-9217-3eec0a4460f7 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:37:46.347305 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-474fec02-246f-4470-9217-3eec0a4460f7 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:37:46.360664 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-474fec02-246f-4470-9217-3eec0a4460f7 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:37:46.362729 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-474fec02-246f-4470-9217-3eec0a4460f7 service neutron] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:37:46.373550 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1172/2375] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 16:37:46 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 43 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 13 16:37:55.150527 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=36 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:37:56.368714 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=51 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:38:33.452214 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a183207e-e8b8-41d4-831d-1065991f2dfd service neutron] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:38:33.470339 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a183207e-e8b8-41d4-831d-1065991f2dfd service neutron] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:38:33.474440 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb12cb32-0845-4544-801d-2b71ceac76e1 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f436f07939d4855a21b195bc7566244', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:38:33.475074 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb12cb32-0845-4544-801d-2b71ceac76e1 service neutron] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:38:33.475351 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb12cb32-0845-4544-801d-2b71ceac76e1 service neutron] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:38:33.475618 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb12cb32-0845-4544-801d-2b71ceac76e1 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:38:33.482634 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb12cb32-0845-4544-801d-2b71ceac76e1 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:38:33.484083 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb12cb32-0845-4544-801d-2b71ceac76e1 service neutron] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:38:33.492796 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1204/2376] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 16:38:33 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 41 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 16:38:43.488323 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=10 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:39:35.398145 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-474fec02-246f-4470-9217-3eec0a4460f7 service neutron] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:39:35.416693 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-474fec02-246f-4470-9217-3eec0a4460f7 service neutron] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:39:35.420843 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4a56104-ca0b-4e53-a7dc-9b5bd427f1ca service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7fd6b5afd67b48f9b0f0597ccd4cf4df', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:39:35.421438 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4a56104-ca0b-4e53-a7dc-9b5bd427f1ca service cinder] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:39:35.421719 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4a56104-ca0b-4e53-a7dc-9b5bd427f1ca service cinder] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:39:35.421997 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4a56104-ca0b-4e53-a7dc-9b5bd427f1ca service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:39:35.430415 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4a56104-ca0b-4e53-a7dc-9b5bd427f1ca service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:39:35.431865 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4a56104-ca0b-4e53-a7dc-9b5bd427f1ca service cinder] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:39:35.441420 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1173/2377] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 16:39:35 2022] GET /identity/v3/auth/tokens => generated 3268 bytes in 44 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 16:39:45.437262 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=10 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:40:40.483519 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb12cb32-0845-4544-801d-2b71ceac76e1 service neutron] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:40:40.492772 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb12cb32-0845-4544-801d-2b71ceac76e1 service neutron] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:40:40.496359 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c8d8318-b771-4504-97df-38f080e5a58f service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7fd6b5afd67b48f9b0f0597ccd4cf4df', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:40:40.496955 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c8d8318-b771-4504-97df-38f080e5a58f service cinder] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:40:40.497226 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c8d8318-b771-4504-97df-38f080e5a58f service cinder] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:40:40.497502 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c8d8318-b771-4504-97df-38f080e5a58f service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:40:40.505772 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c8d8318-b771-4504-97df-38f080e5a58f service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:40:40.507247 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c8d8318-b771-4504-97df-38f080e5a58f service cinder] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:40:40.534445 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1205/2378] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 16:40:40 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 51 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 16:40:41.729798 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4a56104-ca0b-4e53-a7dc-9b5bd427f1ca service cinder] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:40:41.731555 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-e4a56104-ca0b-4e53-a7dc-9b5bd427f1ca service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:40:41.812081 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4a56104-ca0b-4e53-a7dc-9b5bd427f1ca service cinder] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:40:41.817103 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-965764cb-301e-4c3c-abac-888fc879b5fb service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '31480b61418340bd88583f034f67412e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:40:41.817707 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-965764cb-301e-4c3c-abac-888fc879b5fb service placement] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:40:41.817985 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-965764cb-301e-4c3c-abac-888fc879b5fb service placement] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:40:41.818257 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-965764cb-301e-4c3c-abac-888fc879b5fb service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:40:41.820089 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-965764cb-301e-4c3c-abac-888fc879b5fb service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:40:41.832202 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-965764cb-301e-4c3c-abac-888fc879b5fb service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:40:41.833767 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-965764cb-301e-4c3c-abac-888fc879b5fb service placement] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:40:41.843630 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1174/2379] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 16:40:41 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 114 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 16:40:50.532038 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=13 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:40:51.839301 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=37 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:41:32.475314 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c8d8318-b771-4504-97df-38f080e5a58f service cinder] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:41:32.484245 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c8d8318-b771-4504-97df-38f080e5a58f service cinder] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:41:32.488218 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89c6cb36-eb58-4784-9f99-54fc6ff6f500 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7fd6b5afd67b48f9b0f0597ccd4cf4df', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:41:32.488820 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89c6cb36-eb58-4784-9f99-54fc6ff6f500 service cinder] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:41:32.489094 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89c6cb36-eb58-4784-9f99-54fc6ff6f500 service cinder] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:41:32.489363 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89c6cb36-eb58-4784-9f99-54fc6ff6f500 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:41:32.491255 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-89c6cb36-eb58-4784-9f99-54fc6ff6f500 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:41:32.527836 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89c6cb36-eb58-4784-9f99-54fc6ff6f500 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:41:32.529243 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89c6cb36-eb58-4784-9f99-54fc6ff6f500 service cinder] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:41:32.547999 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1206/2380] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 16:41:32 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 73 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 16:41:34.913349 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-965764cb-301e-4c3c-abac-888fc879b5fb service placement] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:41:34.926804 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-965764cb-301e-4c3c-abac-888fc879b5fb service placement] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:41:34.932429 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48486466-b4dc-476b-8330-5882945c9532 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4e039b46c5e3445bbb39904b00c5430e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'admin', 'ResellerAdmin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:41:34.933051 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48486466-b4dc-476b-8330-5882945c9532 service nova] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:41:34.933338 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48486466-b4dc-476b-8330-5882945c9532 service nova] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:41:34.933617 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48486466-b4dc-476b-8330-5882945c9532 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:41:34.935611 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-48486466-b4dc-476b-8330-5882945c9532 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:41:34.949753 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48486466-b4dc-476b-8330-5882945c9532 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:41:34.951555 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48486466-b4dc-476b-8330-5882945c9532 service nova] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:41:34.963751 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1175/2381] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 16:41:34 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 51 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 16:41:35.433567 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89c6cb36-eb58-4784-9f99-54fc6ff6f500 service cinder] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:41:35.448865 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89c6cb36-eb58-4784-9f99-54fc6ff6f500 service cinder] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:41:35.458642 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d510c48e-50d1-44d1-ac26-050735b55164 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f436f07939d4855a21b195bc7566244', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:41:35.459298 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d510c48e-50d1-44d1-ac26-050735b55164 service neutron] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:41:35.460185 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d510c48e-50d1-44d1-ac26-050735b55164 service neutron] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:41:35.460707 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d510c48e-50d1-44d1-ac26-050735b55164 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:41:35.464310 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-d510c48e-50d1-44d1-ac26-050735b55164 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:41:35.653487 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d510c48e-50d1-44d1-ac26-050735b55164 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:41:35.655907 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d510c48e-50d1-44d1-ac26-050735b55164 service neutron] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:41:35.688016 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1207/2382] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 16:41:35 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 255 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 16:41:36.902066 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48486466-b4dc-476b-8330-5882945c9532 service nova] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:41:36.917026 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48486466-b4dc-476b-8330-5882945c9532 service nova] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:41:36.921903 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbd7cf84-fb7d-4f67-acec-c2c10db51706 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f436f07939d4855a21b195bc7566244', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:41:36.922472 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbd7cf84-fb7d-4f67-acec-c2c10db51706 service neutron] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:41:36.922775 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbd7cf84-fb7d-4f67-acec-c2c10db51706 service neutron] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:41:36.923706 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbd7cf84-fb7d-4f67-acec-c2c10db51706 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:41:36.926529 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-bbd7cf84-fb7d-4f67-acec-c2c10db51706 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:41:36.943517 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbd7cf84-fb7d-4f67-acec-c2c10db51706 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:41:36.945038 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbd7cf84-fb7d-4f67-acec-c2c10db51706 service neutron] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:41:36.960076 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1176/2383] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 16:41:36 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 58 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 16:41:37.833116 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d510c48e-50d1-44d1-ac26-050735b55164 service neutron] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:41:37.846327 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d510c48e-50d1-44d1-ac26-050735b55164 service neutron] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:41:37.854944 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-991f387d-04d6-47c5-9fbe-3adbef13e511 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f436f07939d4855a21b195bc7566244', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:41:37.855686 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-991f387d-04d6-47c5-9fbe-3adbef13e511 service neutron] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:41:37.856081 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-991f387d-04d6-47c5-9fbe-3adbef13e511 service neutron] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:41:37.856443 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-991f387d-04d6-47c5-9fbe-3adbef13e511 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:41:37.862370 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-991f387d-04d6-47c5-9fbe-3adbef13e511 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:41:37.900922 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-991f387d-04d6-47c5-9fbe-3adbef13e511 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:41:37.903477 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-991f387d-04d6-47c5-9fbe-3adbef13e511 service neutron] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:41:37.931580 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1208/2384] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 16:41:37 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 99 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 16:41:39.490987 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbd7cf84-fb7d-4f67-acec-c2c10db51706 service neutron] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:41:39.511667 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbd7cf84-fb7d-4f67-acec-c2c10db51706 service neutron] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:41:39.523802 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41b8039b-afea-4df2-be8c-039a97090717 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '31480b61418340bd88583f034f67412e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:41:39.524664 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41b8039b-afea-4df2-be8c-039a97090717 service placement] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:41:39.525090 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41b8039b-afea-4df2-be8c-039a97090717 service placement] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:41:39.525498 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41b8039b-afea-4df2-be8c-039a97090717 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:41:39.527931 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-41b8039b-afea-4df2-be8c-039a97090717 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:41:39.566941 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41b8039b-afea-4df2-be8c-039a97090717 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:41:39.568931 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41b8039b-afea-4df2-be8c-039a97090717 service placement] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:41:39.582441 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1177/2385] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 16:41:39 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 92 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 16:41:39.631162 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-991f387d-04d6-47c5-9fbe-3adbef13e511 service neutron] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:41:39.633168 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-991f387d-04d6-47c5-9fbe-3adbef13e511 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:41:39.654692 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-991f387d-04d6-47c5-9fbe-3adbef13e511 service neutron] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:41:39.662482 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddc9ac5f-d2ff-40a4-b00f-5c199d957fa6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:41:39.665647 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddc9ac5f-d2ff-40a4-b00f-5c199d957fa6 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:41:39.666075 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddc9ac5f-d2ff-40a4-b00f-5c199d957fa6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:41:39.666465 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddc9ac5f-d2ff-40a4-b00f-5c199d957fa6 admin admin] PATH_INFO: `/v3/users/313ac02befbf44ebabf3b41f84245c1f` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:41:39.668102 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ddc9ac5f-d2ff-40a4-b00f-5c199d957fa6 admin admin] RBAC: Authorizing `identity:delete_user(user_id=313ac02befbf44ebabf3b41f84245c1f)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:41:39.670431 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ddc9ac5f-d2ff-40a4-b00f-5c199d957fa6 admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:41:39.762227 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-ddc9ac5f-d2ff-40a4-b00f-5c199d957fa6 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '313ac02befbf44ebabf3b41f84245c1f'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:41:39.776900 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-ddc9ac5f-d2ff-40a4-b00f-5c199d957fa6 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '313ac02befbf44ebabf3b41f84245c1f'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:41:39.840751 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=90,keystone:DELETE=9,keystone:INSERT=1 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:41:39.857157 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-ddc9ac5f-d2ff-40a4-b00f-5c199d957fa6 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '313ac02befbf44ebabf3b41f84245c1f'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:41:39.859349 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-ddc9ac5f-d2ff-40a4-b00f-5c199d957fa6 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '313ac02befbf44ebabf3b41f84245c1f'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:41:39.890788 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-ddc9ac5f-d2ff-40a4-b00f-5c199d957fa6 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '313ac02befbf44ebabf3b41f84245c1f'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:41:39.905387 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1209/2386] 10.222.0.50 () {58 vars in 1224 bytes} [Tue Sep 13 16:41:39 2022] DELETE /identity/v3/users/313ac02befbf44ebabf3b41f84245c1f => generated 0 bytes in 275 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 16:41:39.971347 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41b8039b-afea-4df2-be8c-039a97090717 service placement] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:41:39.998428 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41b8039b-afea-4df2-be8c-039a97090717 service placement] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:41:40.069119 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f74e1ecd-2eab-4503-847d-8304ffc6a941 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f436f07939d4855a21b195bc7566244', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:41:40.069885 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f74e1ecd-2eab-4503-847d-8304ffc6a941 service neutron] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:41:40.070169 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f74e1ecd-2eab-4503-847d-8304ffc6a941 service neutron] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:41:40.070603 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f74e1ecd-2eab-4503-847d-8304ffc6a941 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:41:40.073093 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-f74e1ecd-2eab-4503-847d-8304ffc6a941 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:41:40.191445 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f74e1ecd-2eab-4503-847d-8304ffc6a941 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:41:40.195104 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f74e1ecd-2eab-4503-847d-8304ffc6a941 service neutron] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:41:40.210607 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1178/2387] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 16:41:39 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 240 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 16:41:40.686543 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddc9ac5f-d2ff-40a4-b00f-5c199d957fa6 admin admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:41:40.688549 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-ddc9ac5f-d2ff-40a4-b00f-5c199d957fa6 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:41:40.733387 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f74e1ecd-2eab-4503-847d-8304ffc6a941 service neutron] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:41:40.736981 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-f74e1ecd-2eab-4503-847d-8304ffc6a941 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:41:40.759836 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f74e1ecd-2eab-4503-847d-8304ffc6a941 service neutron] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:41:40.775628 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f10734c-e842-4bc8-a3c6-1f568ee7ac3a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:41:40.776753 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f10734c-e842-4bc8-a3c6-1f568ee7ac3a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:41:40.778452 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f10734c-e842-4bc8-a3c6-1f568ee7ac3a admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:41:40.778837 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f10734c-e842-4bc8-a3c6-1f568ee7ac3a admin admin] PATH_INFO: `/v3/projects/673d53e0757746708c6ca7b9d947b409` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:41:40.782044 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f10734c-e842-4bc8-a3c6-1f568ee7ac3a admin admin] RBAC: Authorizing `identity:delete_project(project_id=673d53e0757746708c6ca7b9d947b409)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:41:40.790146 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f10734c-e842-4bc8-a3c6-1f568ee7ac3a admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:41:40.825104 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddc9ac5f-d2ff-40a4-b00f-5c199d957fa6 admin admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:41:40.831887 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19c33fda-3252-4617-84ea-97d6fc7dd187 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '31480b61418340bd88583f034f67412e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:41:40.832936 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19c33fda-3252-4617-84ea-97d6fc7dd187 service placement] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:41:40.833355 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19c33fda-3252-4617-84ea-97d6fc7dd187 service placement] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:41:40.833838 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19c33fda-3252-4617-84ea-97d6fc7dd187 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:41:40.835996 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-19c33fda-3252-4617-84ea-97d6fc7dd187 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:41:40.854097 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19c33fda-3252-4617-84ea-97d6fc7dd187 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:41:40.856143 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19c33fda-3252-4617-84ea-97d6fc7dd187 service placement] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:41:40.868780 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1210/2388] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 16:41:40 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 183 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 16:41:40.869984 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19c33fda-3252-4617-84ea-97d6fc7dd187 service placement] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:41:40.871698 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-19c33fda-3252-4617-84ea-97d6fc7dd187 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:41:40.883234 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-9f10734c-e842-4bc8-a3c6-1f568ee7ac3a admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '673d53e0757746708c6ca7b9d947b409', 'request_id': 'req-9f10734c-e842-4bc8-a3c6-1f568ee7ac3a'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:41:40.888505 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19c33fda-3252-4617-84ea-97d6fc7dd187 service placement] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:41:40.894914 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-9f10734c-e842-4bc8-a3c6-1f568ee7ac3a admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '673d53e0757746708c6ca7b9d947b409', 'request_id': 'req-9f10734c-e842-4bc8-a3c6-1f568ee7ac3a'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:41:40.907254 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-9f10734c-e842-4bc8-a3c6-1f568ee7ac3a admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '673d53e0757746708c6ca7b9d947b409', 'request_id': 'req-9f10734c-e842-4bc8-a3c6-1f568ee7ac3a'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:41:40.908852 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:41:40.932651 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-9f10734c-e842-4bc8-a3c6-1f568ee7ac3a admin admin] The token cache is being invalidate because project 673d53e0757746708c6ca7b9d947b409 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81438) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 13 16:41:40.934474 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-9f10734c-e842-4bc8-a3c6-1f568ee7ac3a admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:41:40.935744 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-9f10734c-e842-4bc8-a3c6-1f568ee7ac3a admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:41:40.946004 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1179/2389] 10.222.0.50 () {58 vars in 1233 bytes} [Tue Sep 13 16:41:40 2022] DELETE /identity/v3/projects/673d53e0757746708c6ca7b9d947b409 => generated 0 bytes in 207 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 13 16:41:40.950333 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-085bf4a6-568a-435a-afcb-16a82b4f1df7 None None] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:41:40.950739 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-085bf4a6-568a-435a-afcb-16a82b4f1df7 None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:41:40.951149 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-085bf4a6-568a-435a-afcb-16a82b4f1df7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:41:41.023262 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a29148ad-1b1c-48e3-beb2-e343248a5de3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:41:41.024325 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a29148ad-1b1c-48e3-beb2-e343248a5de3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:41:41.024914 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a29148ad-1b1c-48e3-beb2-e343248a5de3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:41:41.025437 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a29148ad-1b1c-48e3-beb2-e343248a5de3 admin admin] PATH_INFO: `/v3/users/e1cbafa1d3c244c5906e475494b39d4c` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:41:41.027010 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a29148ad-1b1c-48e3-beb2-e343248a5de3 admin admin] RBAC: Authorizing `identity:delete_user(user_id=e1cbafa1d3c244c5906e475494b39d4c)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:41:41.031150 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a29148ad-1b1c-48e3-beb2-e343248a5de3 admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:41:41.053090 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.auth.core [None req-085bf4a6-568a-435a-afcb-16a82b4f1df7 None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81438) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:41:41.059337 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-085bf4a6-568a-435a-afcb-16a82b4f1df7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:41:41.073914 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1180/2390] 10.222.0.50 () {58 vars in 958 bytes} [Tue Sep 13 16:41:40 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 127 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:41:41.084765 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6be3df2d-0507-40d5-8227-7cd4c9d4585b None None] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:41:41.085054 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6be3df2d-0507-40d5-8227-7cd4c9d4585b None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:41:41.085339 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6be3df2d-0507-40d5-8227-7cd4c9d4585b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:41:41.114467 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-a29148ad-1b1c-48e3-beb2-e343248a5de3 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'e1cbafa1d3c244c5906e475494b39d4c'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:41:41.119242 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.auth.core [None req-6be3df2d-0507-40d5-8227-7cd4c9d4585b None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81438) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:41:41.120811 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-6be3df2d-0507-40d5-8227-7cd4c9d4585b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:41:41.125460 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-a29148ad-1b1c-48e3-beb2-e343248a5de3 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'e1cbafa1d3c244c5906e475494b39d4c'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:41:41.140889 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1181/2391] 10.222.0.50 () {58 vars in 958 bytes} [Tue Sep 13 16:41:41 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 57 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:41:41.147182 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6be3df2d-0507-40d5-8227-7cd4c9d4585b None None] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:41:41.167627 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6be3df2d-0507-40d5-8227-7cd4c9d4585b None None] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:41:41.173690 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-a29148ad-1b1c-48e3-beb2-e343248a5de3 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'e1cbafa1d3c244c5906e475494b39d4c'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:41:41.173847 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee832781-fd1a-4fe6-988b-9944140ad352 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:41:41.174604 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee832781-fd1a-4fe6-988b-9944140ad352 admin admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:41:41.174714 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-a29148ad-1b1c-48e3-beb2-e343248a5de3 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'e1cbafa1d3c244c5906e475494b39d4c'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:41:41.175406 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee832781-fd1a-4fe6-988b-9944140ad352 admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:41:41.175845 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee832781-fd1a-4fe6-988b-9944140ad352 admin admin] PATH_INFO: `/v3/domains` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:41:41.177326 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee832781-fd1a-4fe6-988b-9944140ad352 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81438) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 16:41:41.177952 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee832781-fd1a-4fe6-988b-9944140ad352 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:41:41.179338 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=88,keystone:DELETE=11,keystone:INSERT=1 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:41:41.181394 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee832781-fd1a-4fe6-988b-9944140ad352 admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:41:41.197286 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-a29148ad-1b1c-48e3-beb2-e343248a5de3 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'e1cbafa1d3c244c5906e475494b39d4c'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:41:41.207118 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1182/2392] 10.222.0.50 () {58 vars in 1153 bytes} [Tue Sep 13 16:41:41 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 62 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 16:41:41.217418 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee832781-fd1a-4fe6-988b-9944140ad352 admin admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:41:41.219564 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-ee832781-fd1a-4fe6-988b-9944140ad352 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:41:41.220061 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1211/2393] 10.222.0.50 () {58 vars in 1224 bytes} [Tue Sep 13 16:41:40 2022] DELETE /identity/v3/users/e1cbafa1d3c244c5906e475494b39d4c => generated 0 bytes in 351 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 16:41:41.244590 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee832781-fd1a-4fe6-988b-9944140ad352 admin admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:41:41.358094 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4461a06e-6588-4ccf-8027-f4c49abf663f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:41:41.358779 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4461a06e-6588-4ccf-8027-f4c49abf663f admin admin] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:41:41.359099 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4461a06e-6588-4ccf-8027-f4c49abf663f admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:41:41.359378 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4461a06e-6588-4ccf-8027-f4c49abf663f admin admin] PATH_INFO: `/v3/projects` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:41:41.360148 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4461a06e-6588-4ccf-8027-f4c49abf663f admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:41:41.362771 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4461a06e-6588-4ccf-8027-f4c49abf663f admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:41:41.406526 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1183/2394] 10.222.0.50 () {60 vars in 1153 bytes} [Tue Sep 13 16:41:41 2022] POST /identity/v3/projects => generated 377 bytes in 189 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 16:41:41.416752 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a29148ad-1b1c-48e3-beb2-e343248a5de3 admin admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:41:41.433627 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a29148ad-1b1c-48e3-beb2-e343248a5de3 admin admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:41:41.500669 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd81983e-6162-4e3f-af7d-defd97eba769 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:41:41.502070 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd81983e-6162-4e3f-af7d-defd97eba769 admin admin] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:41:41.502311 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd81983e-6162-4e3f-af7d-defd97eba769 admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:41:41.502683 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd81983e-6162-4e3f-af7d-defd97eba769 admin admin] PATH_INFO: `/v3/users` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:41:41.503449 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd81983e-6162-4e3f-af7d-defd97eba769 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:41:41.505793 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd81983e-6162-4e3f-af7d-defd97eba769 admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:41:41.531510 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1212/2395] 10.222.0.50 () {60 vars in 1144 bytes} [Tue Sep 13 16:41:41 2022] POST /identity/v3/users => generated 289 bytes in 121 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 16:41:41.535817 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4461a06e-6588-4ccf-8027-f4c49abf663f admin admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:41:41.547479 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4461a06e-6588-4ccf-8027-f4c49abf663f admin admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:41:41.553591 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6081d4af-9192-4641-94fa-1306a7b83063 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:41:41.554301 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6081d4af-9192-4641-94fa-1306a7b83063 admin admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:41:41.554686 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6081d4af-9192-4641-94fa-1306a7b83063 admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:41:41.555132 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6081d4af-9192-4641-94fa-1306a7b83063 admin admin] PATH_INFO: `/v3/roles` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:41:41.556036 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6081d4af-9192-4641-94fa-1306a7b83063 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:41:41.557814 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6081d4af-9192-4641-94fa-1306a7b83063 admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:41:41.570277 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1184/2396] 10.222.0.50 () {58 vars in 1122 bytes} [Tue Sep 13 16:41:41 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 16:41:41.574314 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd81983e-6162-4e3f-af7d-defd97eba769 admin admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:41:41.589864 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd81983e-6162-4e3f-af7d-defd97eba769 admin admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:41:41.596516 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b80f036-7ead-4d03-8d7e-42cd54f43860 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:41:41.597294 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b80f036-7ead-4d03-8d7e-42cd54f43860 admin admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:41:41.598467 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b80f036-7ead-4d03-8d7e-42cd54f43860 admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:41:41.598926 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b80f036-7ead-4d03-8d7e-42cd54f43860 admin admin] PATH_INFO: `/v3/roles` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:41:41.599784 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b80f036-7ead-4d03-8d7e-42cd54f43860 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:41:41.601921 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b80f036-7ead-4d03-8d7e-42cd54f43860 admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:41:41.618158 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1213/2397] 10.222.0.50 () {58 vars in 1122 bytes} [Tue Sep 13 16:41:41 2022] GET /identity/v3/roles => generated 1420 bytes in 44 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 16:41:41.622303 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6081d4af-9192-4641-94fa-1306a7b83063 admin admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:41:41.633714 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b80f036-7ead-4d03-8d7e-42cd54f43860 admin admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:41:41.636955 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6081d4af-9192-4641-94fa-1306a7b83063 admin admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:41:41.643183 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-2b80f036-7ead-4d03-8d7e-42cd54f43860 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:41:41.662260 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fecfd54c-1650-41a1-a629-8ae1da45cb0f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:41:41.662967 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fecfd54c-1650-41a1-a629-8ae1da45cb0f admin admin] REQUEST_METHOD: `PUT` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:41:41.663252 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fecfd54c-1650-41a1-a629-8ae1da45cb0f admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:41:41.663528 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fecfd54c-1650-41a1-a629-8ae1da45cb0f admin admin] PATH_INFO: `/v3/projects/02ef20a5c40945c688b9343861cadb85/users/b6417e189b0d467480627cfd38331a9f/roles/15db4aaca3264ae29bde38903ba43471` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:41:41.693941 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b80f036-7ead-4d03-8d7e-42cd54f43860 admin admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:41:41.703453 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-673dda82-56a8-4e09-b83f-21fb9faf1e03 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:41:41.707555 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-673dda82-56a8-4e09-b83f-21fb9faf1e03 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:41:41.708003 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-673dda82-56a8-4e09-b83f-21fb9faf1e03 admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:41:41.708416 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-673dda82-56a8-4e09-b83f-21fb9faf1e03 admin admin] PATH_INFO: `/v3/projects/4eb7191b18cc41de8c9c51672246991d` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:41:41.709928 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-673dda82-56a8-4e09-b83f-21fb9faf1e03 admin admin] RBAC: Authorizing `identity:delete_project(project_id=4eb7191b18cc41de8c9c51672246991d)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:41:41.719852 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-673dda82-56a8-4e09-b83f-21fb9faf1e03 admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:41:41.720205 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fecfd54c-1650-41a1-a629-8ae1da45cb0f admin admin] RBAC: Authorizing `identity:create_grant(project_id=02ef20a5c40945c688b9343861cadb85, user_id=b6417e189b0d467480627cfd38331a9f, role_id=15db4aaca3264ae29bde38903ba43471)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:41:41.725040 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fecfd54c-1650-41a1-a629-8ae1da45cb0f admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:41:41.754411 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1185/2398] 10.222.0.50 () {60 vars in 1483 bytes} [Tue Sep 13 16:41:41 2022] PUT /identity/v3/projects/02ef20a5c40945c688b9343861cadb85/users/b6417e189b0d467480627cfd38331a9f/roles/15db4aaca3264ae29bde38903ba43471 => generated 0 bytes in 133 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 13 16:41:41.761574 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1e846b7-a0c1-4043-9b7b-55cd45b87bd3 None None] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:41:41.762100 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1e846b7-a0c1-4043-9b7b-55cd45b87bd3 None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:41:41.762539 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1e846b7-a0c1-4043-9b7b-55cd45b87bd3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:41:41.789511 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.auth.core [None req-a1e846b7-a0c1-4043-9b7b-55cd45b87bd3 None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81438) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:41:41.791093 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-a1e846b7-a0c1-4043-9b7b-55cd45b87bd3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:41:41.836714 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:41:41.842858 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-673dda82-56a8-4e09-b83f-21fb9faf1e03 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '4eb7191b18cc41de8c9c51672246991d', 'request_id': 'req-673dda82-56a8-4e09-b83f-21fb9faf1e03'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:41:41.855694 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-673dda82-56a8-4e09-b83f-21fb9faf1e03 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '4eb7191b18cc41de8c9c51672246991d', 'request_id': 'req-673dda82-56a8-4e09-b83f-21fb9faf1e03'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:41:41.879555 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-673dda82-56a8-4e09-b83f-21fb9faf1e03 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '4eb7191b18cc41de8c9c51672246991d', 'request_id': 'req-673dda82-56a8-4e09-b83f-21fb9faf1e03'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:41:41.886486 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1186/2399] 10.222.0.50 () {58 vars in 958 bytes} [Tue Sep 13 16:41:41 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 126 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:41:41.933126 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-673dda82-56a8-4e09-b83f-21fb9faf1e03 admin admin] The token cache is being invalidate because project 4eb7191b18cc41de8c9c51672246991d was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81437) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 13 16:41:41.934030 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-673dda82-56a8-4e09-b83f-21fb9faf1e03 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:41:41.935117 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-673dda82-56a8-4e09-b83f-21fb9faf1e03 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:41:41.937044 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1214/2400] 10.222.0.50 () {58 vars in 1233 bytes} [Tue Sep 13 16:41:41 2022] DELETE /identity/v3/projects/4eb7191b18cc41de8c9c51672246991d => generated 0 bytes in 304 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 16:41:41.956005 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-980d58af-8837-47cb-8803-846ba86c6ceb None None] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:41:41.956289 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-980d58af-8837-47cb-8803-846ba86c6ceb None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:41:41.956558 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-980d58af-8837-47cb-8803-846ba86c6ceb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:41:41.959992 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-673dda82-56a8-4e09-b83f-21fb9faf1e03 admin admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:41:41.962102 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-673dda82-56a8-4e09-b83f-21fb9faf1e03 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:41:41.998614 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.auth.core [None req-980d58af-8837-47cb-8803-846ba86c6ceb None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81438) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:41:42.000149 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-980d58af-8837-47cb-8803-846ba86c6ceb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:41:42.100836 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-673dda82-56a8-4e09-b83f-21fb9faf1e03 admin admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:41:42.105735 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddb9867d-6106-44ad-8655-a87820ccf7b4 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f436f07939d4855a21b195bc7566244', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:41:42.106771 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddb9867d-6106-44ad-8655-a87820ccf7b4 service neutron] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:41:42.107258 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddb9867d-6106-44ad-8655-a87820ccf7b4 service neutron] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:41:42.108262 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddb9867d-6106-44ad-8655-a87820ccf7b4 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:41:42.110041 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-ddb9867d-6106-44ad-8655-a87820ccf7b4 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:41:42.121002 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=89,keystone:INSERT=4,keystone:DELETE=7 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:41:42.156325 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1187/2401] 10.222.0.50 () {58 vars in 958 bytes} [Tue Sep 13 16:41:41 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 202 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:41:42.158389 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e5148f4-c63a-495a-b0b7-b24da693b204 None None] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:41:42.159238 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e5148f4-c63a-495a-b0b7-b24da693b204 None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:41:42.159714 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e5148f4-c63a-495a-b0b7-b24da693b204 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:41:42.200972 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.auth.core [None req-1e5148f4-c63a-495a-b0b7-b24da693b204 None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81438) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:41:42.202470 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-1e5148f4-c63a-495a-b0b7-b24da693b204 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:41:42.223954 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1188/2402] 10.222.0.50 () {58 vars in 958 bytes} [Tue Sep 13 16:41:42 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 67 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:41:42.225113 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e5148f4-c63a-495a-b0b7-b24da693b204 None None] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:41:42.238553 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e5148f4-c63a-495a-b0b7-b24da693b204 None None] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:41:42.250207 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1897f42-fbc4-41ef-add0-ee53957e601d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:41:42.251833 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1897f42-fbc4-41ef-add0-ee53957e601d admin admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:41:42.252486 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1897f42-fbc4-41ef-add0-ee53957e601d admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:41:42.252887 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1897f42-fbc4-41ef-add0-ee53957e601d admin admin] PATH_INFO: `/v3/domains` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:41:42.254931 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1897f42-fbc4-41ef-add0-ee53957e601d admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81438) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 16:41:42.255589 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1897f42-fbc4-41ef-add0-ee53957e601d admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:41:42.257100 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ddb9867d-6106-44ad-8655-a87820ccf7b4 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:41:42.261362 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ddb9867d-6106-44ad-8655-a87820ccf7b4 service neutron] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:41:42.263070 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1897f42-fbc4-41ef-add0-ee53957e601d admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:41:42.291798 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1215/2403] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 16:41:41 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 332 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 16:41:42.300510 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddb9867d-6106-44ad-8655-a87820ccf7b4 service neutron] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:41:42.301799 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1189/2404] 10.222.0.50 () {58 vars in 1153 bytes} [Tue Sep 13 16:41:42 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 77 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 16:41:42.316003 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddb9867d-6106-44ad-8655-a87820ccf7b4 service neutron] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:41:42.323624 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef3422fe-7bbe-401b-bc87-482a3627bfe8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:41:42.325175 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef3422fe-7bbe-401b-bc87-482a3627bfe8 admin admin] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:41:42.325453 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef3422fe-7bbe-401b-bc87-482a3627bfe8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:41:42.325724 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef3422fe-7bbe-401b-bc87-482a3627bfe8 admin admin] PATH_INFO: `/v3/projects` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:41:42.326417 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef3422fe-7bbe-401b-bc87-482a3627bfe8 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:41:42.329057 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef3422fe-7bbe-401b-bc87-482a3627bfe8 admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:41:42.396161 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1216/2405] 10.222.0.50 () {60 vars in 1153 bytes} [Tue Sep 13 16:41:42 2022] POST /identity/v3/projects => generated 369 bytes in 97 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 16:41:42.399872 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1897f42-fbc4-41ef-add0-ee53957e601d admin admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:41:42.411243 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1897f42-fbc4-41ef-add0-ee53957e601d admin admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:41:42.493825 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ef8ae52-eb86-4bb4-a6cc-7d86aa86930a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:41:42.494451 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ef8ae52-eb86-4bb4-a6cc-7d86aa86930a admin admin] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:41:42.494779 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ef8ae52-eb86-4bb4-a6cc-7d86aa86930a admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:41:42.495083 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ef8ae52-eb86-4bb4-a6cc-7d86aa86930a admin admin] PATH_INFO: `/v3/users` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:41:42.496045 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ef8ae52-eb86-4bb4-a6cc-7d86aa86930a admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:41:42.497887 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ef8ae52-eb86-4bb4-a6cc-7d86aa86930a admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:41:42.536263 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1190/2406] 10.222.0.50 () {60 vars in 1144 bytes} [Tue Sep 13 16:41:42 2022] POST /identity/v3/users => generated 285 bytes in 135 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 16:41:42.538248 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef3422fe-7bbe-401b-bc87-482a3627bfe8 admin admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:41:42.556772 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef3422fe-7bbe-401b-bc87-482a3627bfe8 admin admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:41:42.563768 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3197fb52-6286-46cc-a969-fcd012b9852f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:41:42.565129 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3197fb52-6286-46cc-a969-fcd012b9852f admin admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:41:42.565583 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3197fb52-6286-46cc-a969-fcd012b9852f admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:41:42.566059 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3197fb52-6286-46cc-a969-fcd012b9852f admin admin] PATH_INFO: `/v3/roles` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:41:42.566773 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3197fb52-6286-46cc-a969-fcd012b9852f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:41:42.569161 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3197fb52-6286-46cc-a969-fcd012b9852f admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:41:42.590294 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1217/2407] 10.222.0.50 () {58 vars in 1122 bytes} [Tue Sep 13 16:41:42 2022] GET /identity/v3/roles => generated 1420 bytes in 52 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 16:41:42.590511 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ef8ae52-eb86-4bb4-a6cc-7d86aa86930a admin admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:41:42.604611 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ef8ae52-eb86-4bb4-a6cc-7d86aa86930a admin admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:41:42.615561 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0a4b2c1-3140-4bee-b713-b09dc0d6b035 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:41:42.616813 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0a4b2c1-3140-4bee-b713-b09dc0d6b035 admin admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:41:42.617512 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0a4b2c1-3140-4bee-b713-b09dc0d6b035 admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:41:42.617512 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0a4b2c1-3140-4bee-b713-b09dc0d6b035 admin admin] PATH_INFO: `/v3/roles` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:41:42.618141 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0a4b2c1-3140-4bee-b713-b09dc0d6b035 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:41:42.619796 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0a4b2c1-3140-4bee-b713-b09dc0d6b035 admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:41:42.638952 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3197fb52-6286-46cc-a969-fcd012b9852f admin admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:41:42.643067 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1191/2408] 10.222.0.50 () {58 vars in 1122 bytes} [Tue Sep 13 16:41:42 2022] GET /identity/v3/roles => generated 1420 bytes in 53 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 13 16:41:42.651230 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3197fb52-6286-46cc-a969-fcd012b9852f admin admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:41:42.656450 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b34570a9-820e-49fa-9f8e-bcec1d450d03 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:41:42.657105 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b34570a9-820e-49fa-9f8e-bcec1d450d03 admin admin] REQUEST_METHOD: `PUT` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:41:42.657385 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b34570a9-820e-49fa-9f8e-bcec1d450d03 admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:41:42.657659 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b34570a9-820e-49fa-9f8e-bcec1d450d03 admin admin] PATH_INFO: `/v3/projects/05b6ba0827e348c492e84e80a00edd4c/users/455d451ce5fb496e9fffe22a1ca00a41/roles/15db4aaca3264ae29bde38903ba43471` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:41:42.676446 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b34570a9-820e-49fa-9f8e-bcec1d450d03 admin admin] RBAC: Authorizing `identity:create_grant(project_id=05b6ba0827e348c492e84e80a00edd4c, user_id=455d451ce5fb496e9fffe22a1ca00a41, role_id=15db4aaca3264ae29bde38903ba43471)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:41:42.678032 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b34570a9-820e-49fa-9f8e-bcec1d450d03 admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:41:42.698225 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1218/2409] 10.222.0.50 () {60 vars in 1483 bytes} [Tue Sep 13 16:41:42 2022] PUT /identity/v3/projects/05b6ba0827e348c492e84e80a00edd4c/users/455d451ce5fb496e9fffe22a1ca00a41/roles/15db4aaca3264ae29bde38903ba43471 => generated 0 bytes in 61 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 16:41:42.709080 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c7d395d-2714-4b53-994d-6205439061cc None None] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:41:42.709762 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c7d395d-2714-4b53-994d-6205439061cc None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:41:42.710043 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c7d395d-2714-4b53-994d-6205439061cc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:41:42.744303 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.auth.core [None req-0c7d395d-2714-4b53-994d-6205439061cc None None] MFA Rules not processed for user `455d451ce5fb496e9fffe22a1ca00a41`. Rule list: `[]` (Enabled: `True`). {{(pid=81438) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:41:42.746107 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-0c7d395d-2714-4b53-994d-6205439061cc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:41:42.747396 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b34570a9-820e-49fa-9f8e-bcec1d450d03 admin admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:41:42.751094 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-b34570a9-820e-49fa-9f8e-bcec1d450d03 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:41:42.801892 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:41:42.876392 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1192/2410] 10.222.0.50 () {58 vars in 958 bytes} [Tue Sep 13 16:41:42 2022] POST /identity/v3/auth/tokens => generated 3246 bytes in 165 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:41:42.878633 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c7d395d-2714-4b53-994d-6205439061cc None None] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:41:42.886705 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b34570a9-820e-49fa-9f8e-bcec1d450d03 admin admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:41:42.897293 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c7d395d-2714-4b53-994d-6205439061cc None None] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:41:42.898351 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fb9f552-b149-4e85-b3d3-8df25f0d8bcb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:41:42.899032 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fb9f552-b149-4e85-b3d3-8df25f0d8bcb admin admin] REQUEST_METHOD: `DELETE` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:41:42.899314 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fb9f552-b149-4e85-b3d3-8df25f0d8bcb admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:41:42.899593 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fb9f552-b149-4e85-b3d3-8df25f0d8bcb admin admin] PATH_INFO: `/v3/users/77c7141135ae4b8fb3c4cbc3fa153687` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:41:42.902733 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4fb9f552-b149-4e85-b3d3-8df25f0d8bcb admin admin] RBAC: Authorizing `identity:delete_user(user_id=77c7141135ae4b8fb3c4cbc3fa153687)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:41:42.904786 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4fb9f552-b149-4e85-b3d3-8df25f0d8bcb admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:41:42.905231 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a503517-350f-4b78-96e2-487ee7333317 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:41:42.905992 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a503517-350f-4b78-96e2-487ee7333317 admin admin] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:41:42.906283 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a503517-350f-4b78-96e2-487ee7333317 admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:41:42.906562 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a503517-350f-4b78-96e2-487ee7333317 admin admin] PATH_INFO: `/v3/projects` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:41:42.907294 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a503517-350f-4b78-96e2-487ee7333317 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:41:42.910714 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a503517-350f-4b78-96e2-487ee7333317 admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:41:42.967661 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1193/2411] 10.222.0.50 () {60 vars in 1153 bytes} [Tue Sep 13 16:41:42 2022] POST /identity/v3/projects => generated 367 bytes in 87 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 16:41:42.969219 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a503517-350f-4b78-96e2-487ee7333317 admin admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:41:43.012410 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-4fb9f552-b149-4e85-b3d3-8df25f0d8bcb admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '77c7141135ae4b8fb3c4cbc3fa153687'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:41:43.014552 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=91,keystone:INSERT=2,keystone:DELETE=7 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:41:43.026574 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-4fb9f552-b149-4e85-b3d3-8df25f0d8bcb admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '77c7141135ae4b8fb3c4cbc3fa153687'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:41:43.065812 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-4fb9f552-b149-4e85-b3d3-8df25f0d8bcb admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '77c7141135ae4b8fb3c4cbc3fa153687'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:41:43.071435 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-4fb9f552-b149-4e85-b3d3-8df25f0d8bcb admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '77c7141135ae4b8fb3c4cbc3fa153687'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:41:43.084399 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-4fb9f552-b149-4e85-b3d3-8df25f0d8bcb admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '77c7141135ae4b8fb3c4cbc3fa153687'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:41:43.094841 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1219/2412] 10.222.0.50 () {58 vars in 1224 bytes} [Tue Sep 13 16:41:42 2022] DELETE /identity/v3/users/77c7141135ae4b8fb3c4cbc3fa153687 => generated 0 bytes in 349 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 16:41:43.117467 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a503517-350f-4b78-96e2-487ee7333317 admin admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:41:43.204070 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3bd5c1c-2219-4dfb-996e-7b81c95e01b5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:41:43.205271 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3bd5c1c-2219-4dfb-996e-7b81c95e01b5 admin admin] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:41:43.205588 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3bd5c1c-2219-4dfb-996e-7b81c95e01b5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:41:43.205965 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3bd5c1c-2219-4dfb-996e-7b81c95e01b5 admin admin] PATH_INFO: `/v3/users` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:41:43.207165 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3bd5c1c-2219-4dfb-996e-7b81c95e01b5 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:41:43.209516 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3bd5c1c-2219-4dfb-996e-7b81c95e01b5 admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:41:43.244079 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1194/2413] 10.222.0.50 () {60 vars in 1144 bytes} [Tue Sep 13 16:41:42 2022] POST /identity/v3/users => generated 290 bytes in 274 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 16:41:43.245974 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fb9f552-b149-4e85-b3d3-8df25f0d8bcb admin admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:41:43.247899 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-4fb9f552-b149-4e85-b3d3-8df25f0d8bcb admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:41:43.266991 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fb9f552-b149-4e85-b3d3-8df25f0d8bcb admin admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:41:43.276056 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6b16583-2985-4338-bd89-8612ac5f9b52 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:41:43.278186 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6b16583-2985-4338-bd89-8612ac5f9b52 admin admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:41:43.278744 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6b16583-2985-4338-bd89-8612ac5f9b52 admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:41:43.279158 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6b16583-2985-4338-bd89-8612ac5f9b52 admin admin] PATH_INFO: `/v3/roles` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:41:43.280729 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6b16583-2985-4338-bd89-8612ac5f9b52 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:41:43.282411 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6b16583-2985-4338-bd89-8612ac5f9b52 admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:41:43.301001 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1220/2414] 10.222.0.50 () {58 vars in 1122 bytes} [Tue Sep 13 16:41:43 2022] GET /identity/v3/roles => generated 1420 bytes in 56 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 16:41:43.305272 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3bd5c1c-2219-4dfb-996e-7b81c95e01b5 admin admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:41:43.318265 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3bd5c1c-2219-4dfb-996e-7b81c95e01b5 admin admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:41:43.324443 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-806175d0-f24c-444c-a53b-0622bd5a40bb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:41:43.325692 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-806175d0-f24c-444c-a53b-0622bd5a40bb admin admin] REQUEST_METHOD: `PUT` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:41:43.326188 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-806175d0-f24c-444c-a53b-0622bd5a40bb admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:41:43.326701 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-806175d0-f24c-444c-a53b-0622bd5a40bb admin admin] PATH_INFO: `/v3/projects/4bbbc6e825974a71b82562c8544dfa47/users/f464a6c33ea24c7380ba14041b233901/roles/031d4e1a23e34f71858fc4a6dfbd4ccb` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:41:43.348037 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-806175d0-f24c-444c-a53b-0622bd5a40bb admin admin] RBAC: Authorizing `identity:create_grant(project_id=4bbbc6e825974a71b82562c8544dfa47, user_id=f464a6c33ea24c7380ba14041b233901, role_id=031d4e1a23e34f71858fc4a6dfbd4ccb)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:41:43.351245 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-806175d0-f24c-444c-a53b-0622bd5a40bb admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:41:43.374893 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1195/2415] 10.222.0.50 () {60 vars in 1483 bytes} [Tue Sep 13 16:41:43 2022] PUT /identity/v3/projects/4bbbc6e825974a71b82562c8544dfa47/users/f464a6c33ea24c7380ba14041b233901/roles/031d4e1a23e34f71858fc4a6dfbd4ccb => generated 0 bytes in 70 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 16:41:43.378791 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6b16583-2985-4338-bd89-8612ac5f9b52 admin admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:41:43.393435 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6b16583-2985-4338-bd89-8612ac5f9b52 admin admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:41:43.503566 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b77f8ddb-1643-4494-82ae-dfb9f50d8084 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:41:43.507544 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b77f8ddb-1643-4494-82ae-dfb9f50d8084 admin admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:41:43.516371 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b77f8ddb-1643-4494-82ae-dfb9f50d8084 admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:41:43.516747 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b77f8ddb-1643-4494-82ae-dfb9f50d8084 admin admin] PATH_INFO: `/v3/roles` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:41:43.520525 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b77f8ddb-1643-4494-82ae-dfb9f50d8084 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:41:43.522278 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b77f8ddb-1643-4494-82ae-dfb9f50d8084 admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:41:43.546137 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-806175d0-f24c-444c-a53b-0622bd5a40bb admin admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:41:43.548256 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-806175d0-f24c-444c-a53b-0622bd5a40bb admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:41:43.556637 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1221/2416] 10.222.0.50 () {58 vars in 1122 bytes} [Tue Sep 13 16:41:43 2022] GET /identity/v3/roles => generated 1420 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 16:41:43.557762 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b77f8ddb-1643-4494-82ae-dfb9f50d8084 admin admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:41:43.584713 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b77f8ddb-1643-4494-82ae-dfb9f50d8084 admin admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:41:43.590788 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-806175d0-f24c-444c-a53b-0622bd5a40bb admin admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:41:43.593146 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d979314-c2fe-421e-9520-1f9c77960d7a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:41:43.594537 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d979314-c2fe-421e-9520-1f9c77960d7a admin admin] REQUEST_METHOD: `PUT` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:41:43.594769 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d979314-c2fe-421e-9520-1f9c77960d7a admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:41:43.595095 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d979314-c2fe-421e-9520-1f9c77960d7a admin admin] PATH_INFO: `/v3/projects/4bbbc6e825974a71b82562c8544dfa47/users/f464a6c33ea24c7380ba14041b233901/roles/031d4e1a23e34f71858fc4a6dfbd4ccb` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:41:43.600186 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d979314-c2fe-421e-9520-1f9c77960d7a admin admin] RBAC: Authorizing `identity:create_grant(project_id=4bbbc6e825974a71b82562c8544dfa47, user_id=f464a6c33ea24c7380ba14041b233901, role_id=031d4e1a23e34f71858fc4a6dfbd4ccb)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:41:43.603400 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d979314-c2fe-421e-9520-1f9c77960d7a admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:41:43.604417 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d158d439-4492-4868-8720-e9fdf7bfcc35 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:41:43.606247 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d158d439-4492-4868-8720-e9fdf7bfcc35 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:41:43.607311 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d158d439-4492-4868-8720-e9fdf7bfcc35 admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:41:43.610166 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d158d439-4492-4868-8720-e9fdf7bfcc35 admin admin] PATH_INFO: `/v3/projects/8fab8596323c484f806904158532bb49` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:41:43.612698 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d158d439-4492-4868-8720-e9fdf7bfcc35 admin admin] RBAC: Authorizing `identity:delete_project(project_id=8fab8596323c484f806904158532bb49)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:41:43.616144 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d158d439-4492-4868-8720-e9fdf7bfcc35 admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:41:43.621192 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1222/2417] 10.222.0.50 () {60 vars in 1483 bytes} [Tue Sep 13 16:41:43 2022] PUT /identity/v3/projects/4bbbc6e825974a71b82562c8544dfa47/users/f464a6c33ea24c7380ba14041b233901/roles/031d4e1a23e34f71858fc4a6dfbd4ccb => generated 0 bytes in 64 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 16:41:43.643539 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88d72b22-6564-4f29-9e07-4dec8967adee None None] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:41:43.643637 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88d72b22-6564-4f29-9e07-4dec8967adee None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:41:43.643939 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88d72b22-6564-4f29-9e07-4dec8967adee None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:41:43.652218 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:41:43.687732 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.auth.core [None req-88d72b22-6564-4f29-9e07-4dec8967adee None None] MFA Rules not processed for user `f464a6c33ea24c7380ba14041b233901`. Rule list: `[]` (Enabled: `True`). {{(pid=81437) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:41:43.689264 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-88d72b22-6564-4f29-9e07-4dec8967adee None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:41:43.742309 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-d158d439-4492-4868-8720-e9fdf7bfcc35 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '8fab8596323c484f806904158532bb49', 'request_id': 'req-d158d439-4492-4868-8720-e9fdf7bfcc35'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:41:43.761991 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-d158d439-4492-4868-8720-e9fdf7bfcc35 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '8fab8596323c484f806904158532bb49', 'request_id': 'req-d158d439-4492-4868-8720-e9fdf7bfcc35'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:41:43.771753 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-d158d439-4492-4868-8720-e9fdf7bfcc35 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '8fab8596323c484f806904158532bb49', 'request_id': 'req-d158d439-4492-4868-8720-e9fdf7bfcc35'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:41:43.788736 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-d158d439-4492-4868-8720-e9fdf7bfcc35 admin admin] The token cache is being invalidate because project 8fab8596323c484f806904158532bb49 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81438) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 13 16:41:43.789052 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-d158d439-4492-4868-8720-e9fdf7bfcc35 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:41:43.789817 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-d158d439-4492-4868-8720-e9fdf7bfcc35 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:41:43.791726 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1196/2418] 10.222.0.50 () {58 vars in 1233 bytes} [Tue Sep 13 16:41:43 2022] DELETE /identity/v3/projects/8fab8596323c484f806904158532bb49 => generated 0 bytes in 246 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 13 16:41:43.801247 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-394cf815-28b1-44ba-8355-2b01719193be None None] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:41:43.801654 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-394cf815-28b1-44ba-8355-2b01719193be None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:41:43.802410 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-394cf815-28b1-44ba-8355-2b01719193be None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:41:43.844024 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.auth.core [None req-394cf815-28b1-44ba-8355-2b01719193be None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81438) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:41:43.846434 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-394cf815-28b1-44ba-8355-2b01719193be None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:41:43.859190 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1223/2419] 10.222.0.50 () {58 vars in 958 bytes} [Tue Sep 13 16:41:43 2022] POST /identity/v3/auth/tokens => generated 3311 bytes in 225 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:41:43.867385 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88d72b22-6564-4f29-9e07-4dec8967adee None None] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:41:43.869601 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-88d72b22-6564-4f29-9e07-4dec8967adee None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:41:43.941458 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:INSERT=2,keystone:DELETE=3,keystone:SELECT=95 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:41:43.986954 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1197/2420] 10.222.0.50 () {58 vars in 958 bytes} [Tue Sep 13 16:41:43 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 185 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:41:43.993016 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb3d753f-23a2-4be3-9d46-3fc15490e500 None None] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:41:43.993788 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb3d753f-23a2-4be3-9d46-3fc15490e500 None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:41:43.994234 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb3d753f-23a2-4be3-9d46-3fc15490e500 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:41:44.022784 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88d72b22-6564-4f29-9e07-4dec8967adee None None] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:41:44.050261 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.auth.core [None req-cb3d753f-23a2-4be3-9d46-3fc15490e500 None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81438) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:41:44.052064 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-cb3d753f-23a2-4be3-9d46-3fc15490e500 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:41:44.078220 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1198/2421] 10.222.0.50 () {58 vars in 958 bytes} [Tue Sep 13 16:41:43 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:41:44.089884 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb3d753f-23a2-4be3-9d46-3fc15490e500 None None] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:41:44.107743 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb3d753f-23a2-4be3-9d46-3fc15490e500 None None] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:41:44.134799 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccfaac7a-cd29-4586-b20b-a277ab9cb1dc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:41:44.135532 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccfaac7a-cd29-4586-b20b-a277ab9cb1dc admin admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:41:44.135815 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccfaac7a-cd29-4586-b20b-a277ab9cb1dc admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:41:44.136095 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccfaac7a-cd29-4586-b20b-a277ab9cb1dc admin admin] PATH_INFO: `/v3/domains` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:41:44.136817 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ccfaac7a-cd29-4586-b20b-a277ab9cb1dc admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81438) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 16:41:44.137131 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ccfaac7a-cd29-4586-b20b-a277ab9cb1dc admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:41:44.138771 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ccfaac7a-cd29-4586-b20b-a277ab9cb1dc admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:41:44.156446 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1199/2422] 10.222.0.50 () {58 vars in 1153 bytes} [Tue Sep 13 16:41:44 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 64 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 16:41:44.159443 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccfaac7a-cd29-4586-b20b-a277ab9cb1dc admin admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:41:44.162586 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-462e3c2e-9e29-4d39-8017-8f50c290654d service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7fd6b5afd67b48f9b0f0597ccd4cf4df', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:41:44.164212 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-462e3c2e-9e29-4d39-8017-8f50c290654d service cinder] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:41:44.164504 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-462e3c2e-9e29-4d39-8017-8f50c290654d service cinder] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:41:44.164783 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-462e3c2e-9e29-4d39-8017-8f50c290654d service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:41:44.185314 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccfaac7a-cd29-4586-b20b-a277ab9cb1dc admin admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:41:44.201519 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc1f57fa-b339-4c72-9d62-9bd8ff2e02a3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:41:44.202148 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc1f57fa-b339-4c72-9d62-9bd8ff2e02a3 admin admin] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:41:44.202435 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc1f57fa-b339-4c72-9d62-9bd8ff2e02a3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:41:44.202715 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc1f57fa-b339-4c72-9d62-9bd8ff2e02a3 admin admin] PATH_INFO: `/v3/projects` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:41:44.203394 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc1f57fa-b339-4c72-9d62-9bd8ff2e02a3 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:41:44.207395 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc1f57fa-b339-4c72-9d62-9bd8ff2e02a3 admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:41:44.245345 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-462e3c2e-9e29-4d39-8017-8f50c290654d service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:41:44.247240 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-462e3c2e-9e29-4d39-8017-8f50c290654d service cinder] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:41:44.262874 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1224/2423] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 16:41:43 2022] GET /identity/v3/auth/tokens => generated 3311 bytes in 396 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 16:41:44.298186 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1200/2424] 10.222.0.50 () {60 vars in 1153 bytes} [Tue Sep 13 16:41:44 2022] POST /identity/v3/projects => generated 363 bytes in 139 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 16:41:44.299485 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-462e3c2e-9e29-4d39-8017-8f50c290654d service cinder] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:41:44.318224 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-462e3c2e-9e29-4d39-8017-8f50c290654d service cinder] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:41:44.462648 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da5d7c9a-41be-4937-8a63-1fecb6117bed admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:41:44.463535 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da5d7c9a-41be-4937-8a63-1fecb6117bed admin admin] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:41:44.463960 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da5d7c9a-41be-4937-8a63-1fecb6117bed admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:41:44.464358 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da5d7c9a-41be-4937-8a63-1fecb6117bed admin admin] PATH_INFO: `/v3/users` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:41:44.465123 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da5d7c9a-41be-4937-8a63-1fecb6117bed admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:41:44.476873 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da5d7c9a-41be-4937-8a63-1fecb6117bed admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:41:44.520075 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1225/2425] 10.222.0.50 () {60 vars in 1144 bytes} [Tue Sep 13 16:41:44 2022] POST /identity/v3/users => generated 282 bytes in 222 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 13 16:41:44.525999 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc1f57fa-b339-4c72-9d62-9bd8ff2e02a3 admin admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:41:44.541894 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc1f57fa-b339-4c72-9d62-9bd8ff2e02a3 admin admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:41:44.552664 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2aa6ac56-8133-4aa1-9ffe-b3e41ccc25ab admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:41:44.554538 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2aa6ac56-8133-4aa1-9ffe-b3e41ccc25ab admin admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:41:44.555719 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2aa6ac56-8133-4aa1-9ffe-b3e41ccc25ab admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:41:44.556586 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2aa6ac56-8133-4aa1-9ffe-b3e41ccc25ab admin admin] PATH_INFO: `/v3/roles` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:41:44.558464 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2aa6ac56-8133-4aa1-9ffe-b3e41ccc25ab admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:41:44.560337 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2aa6ac56-8133-4aa1-9ffe-b3e41ccc25ab admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:41:44.583708 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1201/2426] 10.222.0.50 () {58 vars in 1122 bytes} [Tue Sep 13 16:41:44 2022] GET /identity/v3/roles => generated 1420 bytes in 55 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 16:41:44.585027 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da5d7c9a-41be-4937-8a63-1fecb6117bed admin admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:41:44.600336 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da5d7c9a-41be-4937-8a63-1fecb6117bed admin admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:41:44.610740 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b39bedbb-9ddf-423f-87a3-52e0499ccd75 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:41:44.611596 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b39bedbb-9ddf-423f-87a3-52e0499ccd75 admin admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:41:44.612005 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b39bedbb-9ddf-423f-87a3-52e0499ccd75 admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:41:44.612271 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b39bedbb-9ddf-423f-87a3-52e0499ccd75 admin admin] PATH_INFO: `/v3/roles` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:41:44.613664 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b39bedbb-9ddf-423f-87a3-52e0499ccd75 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:41:44.615602 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b39bedbb-9ddf-423f-87a3-52e0499ccd75 admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:41:44.631772 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1226/2427] 10.222.0.50 () {58 vars in 1122 bytes} [Tue Sep 13 16:41:44 2022] GET /identity/v3/roles => generated 1420 bytes in 44 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 16:41:44.632908 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2aa6ac56-8133-4aa1-9ffe-b3e41ccc25ab admin admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:41:44.650156 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2aa6ac56-8133-4aa1-9ffe-b3e41ccc25ab admin admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:41:44.662335 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfb9f8f6-0325-42ae-886c-f1bf21d74ec8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:41:44.663075 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfb9f8f6-0325-42ae-886c-f1bf21d74ec8 admin admin] REQUEST_METHOD: `PUT` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:41:44.663363 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfb9f8f6-0325-42ae-886c-f1bf21d74ec8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:41:44.663695 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfb9f8f6-0325-42ae-886c-f1bf21d74ec8 admin admin] PATH_INFO: `/v3/projects/e4a3fac5db2443b492844028e5cbee61/users/61b2fd10440e4267a71553f6d6f793a8/roles/15db4aaca3264ae29bde38903ba43471` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:41:44.707622 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dfb9f8f6-0325-42ae-886c-f1bf21d74ec8 admin admin] RBAC: Authorizing `identity:create_grant(project_id=e4a3fac5db2443b492844028e5cbee61, user_id=61b2fd10440e4267a71553f6d6f793a8, role_id=15db4aaca3264ae29bde38903ba43471)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:41:44.717810 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dfb9f8f6-0325-42ae-886c-f1bf21d74ec8 admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:41:44.773226 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1202/2428] 10.222.0.50 () {60 vars in 1483 bytes} [Tue Sep 13 16:41:44 2022] PUT /identity/v3/projects/e4a3fac5db2443b492844028e5cbee61/users/61b2fd10440e4267a71553f6d6f793a8/roles/15db4aaca3264ae29bde38903ba43471 => generated 0 bytes in 131 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 16:41:44.773762 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93174f56-ec80-453a-85d5-0a0188aee64a None None] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:41:44.774055 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93174f56-ec80-453a-85d5-0a0188aee64a None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:41:44.774335 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93174f56-ec80-453a-85d5-0a0188aee64a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:41:44.792441 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:41:44.818806 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.auth.core [None req-93174f56-ec80-453a-85d5-0a0188aee64a None None] MFA Rules not processed for user `61b2fd10440e4267a71553f6d6f793a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81437) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:41:44.820715 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-93174f56-ec80-453a-85d5-0a0188aee64a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:41:44.969708 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1227/2429] 10.222.0.50 () {58 vars in 958 bytes} [Tue Sep 13 16:41:44 2022] POST /identity/v3/auth/tokens => generated 3240 bytes in 197 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:41:44.974597 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfb9f8f6-0325-42ae-886c-f1bf21d74ec8 admin admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:41:45.055329 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:41:45.160105 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfb9f8f6-0325-42ae-886c-f1bf21d74ec8 admin admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:41:45.184901 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1b90c07-782c-455a-aa60-2f4be06c2449 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:41:45.186336 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1b90c07-782c-455a-aa60-2f4be06c2449 admin admin] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:41:45.187280 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1b90c07-782c-455a-aa60-2f4be06c2449 admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:41:45.187933 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1b90c07-782c-455a-aa60-2f4be06c2449 admin admin] PATH_INFO: `/v3/projects` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:41:45.188958 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1b90c07-782c-455a-aa60-2f4be06c2449 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:41:45.192557 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1b90c07-782c-455a-aa60-2f4be06c2449 admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:41:45.253839 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1203/2430] 10.222.0.50 () {60 vars in 1153 bytes} [Tue Sep 13 16:41:44 2022] POST /identity/v3/projects => generated 365 bytes in 277 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 16:41:45.255417 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93174f56-ec80-453a-85d5-0a0188aee64a None None] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:41:45.271506 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93174f56-ec80-453a-85d5-0a0188aee64a None None] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:41:45.397816 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0899025f-fc65-43fc-b06b-b0470425839c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:41:45.398508 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0899025f-fc65-43fc-b06b-b0470425839c admin admin] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:41:45.399156 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0899025f-fc65-43fc-b06b-b0470425839c admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:41:45.399156 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0899025f-fc65-43fc-b06b-b0470425839c admin admin] PATH_INFO: `/v3/users` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:41:45.399847 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0899025f-fc65-43fc-b06b-b0470425839c admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:41:45.403822 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0899025f-fc65-43fc-b06b-b0470425839c admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:41:45.501804 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1228/2431] 10.222.0.50 () {60 vars in 1144 bytes} [Tue Sep 13 16:41:45 2022] POST /identity/v3/users => generated 289 bytes in 247 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 16:41:45.505862 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1b90c07-782c-455a-aa60-2f4be06c2449 admin admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:41:45.530091 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1b90c07-782c-455a-aa60-2f4be06c2449 admin admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:41:45.535628 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f0bbd2f-ab9f-4b2a-ad4c-5199f6b66bee admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:41:45.536252 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f0bbd2f-ab9f-4b2a-ad4c-5199f6b66bee admin admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:41:45.536531 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f0bbd2f-ab9f-4b2a-ad4c-5199f6b66bee admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:41:45.537050 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f0bbd2f-ab9f-4b2a-ad4c-5199f6b66bee admin admin] PATH_INFO: `/v3/roles` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:41:45.537743 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f0bbd2f-ab9f-4b2a-ad4c-5199f6b66bee admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:41:45.539528 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f0bbd2f-ab9f-4b2a-ad4c-5199f6b66bee admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:41:45.557764 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1204/2432] 10.222.0.50 () {58 vars in 1122 bytes} [Tue Sep 13 16:41:45 2022] GET /identity/v3/roles => generated 1420 bytes in 52 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 16:41:45.561512 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0899025f-fc65-43fc-b06b-b0470425839c admin admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:41:45.573864 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0899025f-fc65-43fc-b06b-b0470425839c admin admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:41:45.594290 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-872a3902-40e6-4b0f-8108-5f29106055e1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:41:45.595005 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-872a3902-40e6-4b0f-8108-5f29106055e1 admin admin] REQUEST_METHOD: `PUT` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:41:45.595431 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-872a3902-40e6-4b0f-8108-5f29106055e1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:41:45.595766 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-872a3902-40e6-4b0f-8108-5f29106055e1 admin admin] PATH_INFO: `/v3/projects/aa19ae24f9344604aa94f624eb110bc9/users/de5c6be59405407b989a0e2b3ff08046/roles/031d4e1a23e34f71858fc4a6dfbd4ccb` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:41:45.615698 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-872a3902-40e6-4b0f-8108-5f29106055e1 admin admin] RBAC: Authorizing `identity:create_grant(project_id=aa19ae24f9344604aa94f624eb110bc9, user_id=de5c6be59405407b989a0e2b3ff08046, role_id=031d4e1a23e34f71858fc4a6dfbd4ccb)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:41:45.618454 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-872a3902-40e6-4b0f-8108-5f29106055e1 admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:41:45.670601 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1229/2433] 10.222.0.50 () {60 vars in 1483 bytes} [Tue Sep 13 16:41:45 2022] PUT /identity/v3/projects/aa19ae24f9344604aa94f624eb110bc9/users/de5c6be59405407b989a0e2b3ff08046/roles/031d4e1a23e34f71858fc4a6dfbd4ccb => generated 0 bytes in 109 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 16:41:45.676484 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f0bbd2f-ab9f-4b2a-ad4c-5199f6b66bee admin admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:41:45.694943 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f0bbd2f-ab9f-4b2a-ad4c-5199f6b66bee admin admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:41:45.813117 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e0b927a-9182-4d6d-99b6-6cb08b39e254 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:41:45.813117 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e0b927a-9182-4d6d-99b6-6cb08b39e254 admin admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:41:45.813117 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e0b927a-9182-4d6d-99b6-6cb08b39e254 admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:41:45.813117 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e0b927a-9182-4d6d-99b6-6cb08b39e254 admin admin] PATH_INFO: `/v3/roles` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:41:45.813603 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e0b927a-9182-4d6d-99b6-6cb08b39e254 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:41:45.813603 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e0b927a-9182-4d6d-99b6-6cb08b39e254 admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:41:45.832064 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1205/2434] 10.222.0.50 () {58 vars in 1122 bytes} [Tue Sep 13 16:41:45 2022] GET /identity/v3/roles => generated 1420 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 16:41:45.837679 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-872a3902-40e6-4b0f-8108-5f29106055e1 admin admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:41:45.860473 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-872a3902-40e6-4b0f-8108-5f29106055e1 admin admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:41:45.873622 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e0b927a-9182-4d6d-99b6-6cb08b39e254 admin admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:41:45.877524 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b39307ac-984a-4263-80ad-f32eb769a1a5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:41:45.877779 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-1e0b927a-9182-4d6d-99b6-6cb08b39e254 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:41:45.878381 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b39307ac-984a-4263-80ad-f32eb769a1a5 admin admin] REQUEST_METHOD: `PUT` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:41:45.878739 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b39307ac-984a-4263-80ad-f32eb769a1a5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:41:45.879119 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b39307ac-984a-4263-80ad-f32eb769a1a5 admin admin] PATH_INFO: `/v3/projects/aa19ae24f9344604aa94f624eb110bc9/users/de5c6be59405407b989a0e2b3ff08046/roles/031d4e1a23e34f71858fc4a6dfbd4ccb` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:41:45.882803 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b39307ac-984a-4263-80ad-f32eb769a1a5 admin admin] RBAC: Authorizing `identity:create_grant(project_id=aa19ae24f9344604aa94f624eb110bc9, user_id=de5c6be59405407b989a0e2b3ff08046, role_id=031d4e1a23e34f71858fc4a6dfbd4ccb)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:41:45.886962 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b39307ac-984a-4263-80ad-f32eb769a1a5 admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:41:45.920830 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1230/2435] 10.222.0.50 () {60 vars in 1483 bytes} [Tue Sep 13 16:41:45 2022] PUT /identity/v3/projects/aa19ae24f9344604aa94f624eb110bc9/users/de5c6be59405407b989a0e2b3ff08046/roles/031d4e1a23e34f71858fc4a6dfbd4ccb => generated 0 bytes in 84 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 16:41:45.922714 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e0b927a-9182-4d6d-99b6-6cb08b39e254 admin admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:41:45.933666 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61297b5a-5927-44af-ba69-d1e34546fa50 None None] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:41:45.933955 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61297b5a-5927-44af-ba69-d1e34546fa50 None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:41:45.934240 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61297b5a-5927-44af-ba69-d1e34546fa50 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:41:45.993544 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.auth.core [None req-61297b5a-5927-44af-ba69-d1e34546fa50 None None] MFA Rules not processed for user `de5c6be59405407b989a0e2b3ff08046`. Rule list: `[]` (Enabled: `True`). {{(pid=81437) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:41:45.995665 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-61297b5a-5927-44af-ba69-d1e34546fa50 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:41:46.048834 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:41:46.104366 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9d2ccb7-e5e4-49e0-b2c8-5809a9ccaea0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:41:46.112157 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9d2ccb7-e5e4-49e0-b2c8-5809a9ccaea0 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:41:46.113466 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9d2ccb7-e5e4-49e0-b2c8-5809a9ccaea0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:41:46.114826 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9d2ccb7-e5e4-49e0-b2c8-5809a9ccaea0 admin admin] PATH_INFO: `/v3/users/6d31b2ec391441f7bab3546d2354b0a1` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:41:46.119291 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9d2ccb7-e5e4-49e0-b2c8-5809a9ccaea0 admin admin] RBAC: Authorizing `identity:delete_user(user_id=6d31b2ec391441f7bab3546d2354b0a1)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:41:46.121340 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9d2ccb7-e5e4-49e0-b2c8-5809a9ccaea0 admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:41:46.153334 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1231/2436] 10.222.0.50 () {58 vars in 958 bytes} [Tue Sep 13 16:41:45 2022] POST /identity/v3/auth/tokens => generated 3309 bytes in 220 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:41:46.162021 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61297b5a-5927-44af-ba69-d1e34546fa50 None None] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:41:46.163976 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-61297b5a-5927-44af-ba69-d1e34546fa50 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:41:46.206759 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=94,keystone:INSERT=1,keystone:DELETE=5 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:41:46.228343 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-e9d2ccb7-e5e4-49e0-b2c8-5809a9ccaea0 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '6d31b2ec391441f7bab3546d2354b0a1'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:41:46.239679 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-e9d2ccb7-e5e4-49e0-b2c8-5809a9ccaea0 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '6d31b2ec391441f7bab3546d2354b0a1'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:41:46.272115 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-e9d2ccb7-e5e4-49e0-b2c8-5809a9ccaea0 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '6d31b2ec391441f7bab3546d2354b0a1'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:41:46.273963 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-e9d2ccb7-e5e4-49e0-b2c8-5809a9ccaea0 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '6d31b2ec391441f7bab3546d2354b0a1'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:41:46.284521 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-e9d2ccb7-e5e4-49e0-b2c8-5809a9ccaea0 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '6d31b2ec391441f7bab3546d2354b0a1'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:41:46.303805 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1206/2437] 10.222.0.50 () {58 vars in 1224 bytes} [Tue Sep 13 16:41:45 2022] DELETE /identity/v3/users/6d31b2ec391441f7bab3546d2354b0a1 => generated 0 bytes in 430 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 16:41:46.323199 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61297b5a-5927-44af-ba69-d1e34546fa50 None None] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:41:46.465686 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2444a75-0a3f-420e-ac1e-abc198028d14 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7fd6b5afd67b48f9b0f0597ccd4cf4df', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:41:46.466568 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2444a75-0a3f-420e-ac1e-abc198028d14 service cinder] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:41:46.466864 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2444a75-0a3f-420e-ac1e-abc198028d14 service cinder] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:41:46.467191 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2444a75-0a3f-420e-ac1e-abc198028d14 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:41:46.478482 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-f2444a75-0a3f-420e-ac1e-abc198028d14 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:41:46.635452 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9d2ccb7-e5e4-49e0-b2c8-5809a9ccaea0 admin admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:41:46.637185 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-e9d2ccb7-e5e4-49e0-b2c8-5809a9ccaea0 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:41:46.654629 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2444a75-0a3f-420e-ac1e-abc198028d14 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:41:46.658612 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2444a75-0a3f-420e-ac1e-abc198028d14 service cinder] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:41:46.700029 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1232/2438] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 16:41:46 2022] GET /identity/v3/auth/tokens => generated 3309 bytes in 539 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 13 16:41:46.820638 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9d2ccb7-e5e4-49e0-b2c8-5809a9ccaea0 admin admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:41:46.828026 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2444a75-0a3f-420e-ac1e-abc198028d14 service cinder] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:41:46.829545 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-f2444a75-0a3f-420e-ac1e-abc198028d14 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:41:46.837499 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d25486c-d360-447f-a3a3-e37ab4ed8a08 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:41:46.838162 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d25486c-d360-447f-a3a3-e37ab4ed8a08 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:41:46.838447 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d25486c-d360-447f-a3a3-e37ab4ed8a08 admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:41:46.838952 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d25486c-d360-447f-a3a3-e37ab4ed8a08 admin admin] PATH_INFO: `/v3/projects/5b11b198b8754bcabf244e0047475b8d` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:41:46.841476 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d25486c-d360-447f-a3a3-e37ab4ed8a08 admin admin] RBAC: Authorizing `identity:delete_project(project_id=5b11b198b8754bcabf244e0047475b8d)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:41:46.844230 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d25486c-d360-447f-a3a3-e37ab4ed8a08 admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:41:46.852945 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2444a75-0a3f-420e-ac1e-abc198028d14 service cinder] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:41:46.870389 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c2578b9-d9bb-411f-8c5d-aee3f292b186 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:41:46.871092 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c2578b9-d9bb-411f-8c5d-aee3f292b186 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:41:46.871455 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c2578b9-d9bb-411f-8c5d-aee3f292b186 admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:41:46.871792 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c2578b9-d9bb-411f-8c5d-aee3f292b186 admin admin] PATH_INFO: `/v3/users/455d451ce5fb496e9fffe22a1ca00a41` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:41:46.874380 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c2578b9-d9bb-411f-8c5d-aee3f292b186 admin admin] RBAC: Authorizing `identity:delete_user(user_id=455d451ce5fb496e9fffe22a1ca00a41)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:41:46.876432 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c2578b9-d9bb-411f-8c5d-aee3f292b186 admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:41:46.961513 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-5d25486c-d360-447f-a3a3-e37ab4ed8a08 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '5b11b198b8754bcabf244e0047475b8d', 'request_id': 'req-5d25486c-d360-447f-a3a3-e37ab4ed8a08'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:41:46.964351 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=95,keystone:DELETE=5 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:41:46.978983 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-5d25486c-d360-447f-a3a3-e37ab4ed8a08 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '5b11b198b8754bcabf244e0047475b8d', 'request_id': 'req-5d25486c-d360-447f-a3a3-e37ab4ed8a08'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:41:46.988847 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-2c2578b9-d9bb-411f-8c5d-aee3f292b186 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '455d451ce5fb496e9fffe22a1ca00a41'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:41:46.991526 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-5d25486c-d360-447f-a3a3-e37ab4ed8a08 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '5b11b198b8754bcabf244e0047475b8d', 'request_id': 'req-5d25486c-d360-447f-a3a3-e37ab4ed8a08'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:41:47.004592 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-2c2578b9-d9bb-411f-8c5d-aee3f292b186 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '455d451ce5fb496e9fffe22a1ca00a41'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:41:47.006652 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-5d25486c-d360-447f-a3a3-e37ab4ed8a08 admin admin] The token cache is being invalidate because project 5b11b198b8754bcabf244e0047475b8d was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81438) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 13 16:41:47.007333 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-5d25486c-d360-447f-a3a3-e37ab4ed8a08 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:41:47.008230 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-5d25486c-d360-447f-a3a3-e37ab4ed8a08 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:41:47.010311 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1207/2439] 10.222.0.50 () {58 vars in 1233 bytes} [Tue Sep 13 16:41:46 2022] DELETE /identity/v3/projects/5b11b198b8754bcabf244e0047475b8d => generated 0 bytes in 376 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 13 16:41:47.024061 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cad8fb1f-4a21-402d-8d91-029a29ccf6db None None] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:41:47.024724 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cad8fb1f-4a21-402d-8d91-029a29ccf6db None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:41:47.025206 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cad8fb1f-4a21-402d-8d91-029a29ccf6db None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:41:47.039556 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-2c2578b9-d9bb-411f-8c5d-aee3f292b186 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '455d451ce5fb496e9fffe22a1ca00a41'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:41:47.040325 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-2c2578b9-d9bb-411f-8c5d-aee3f292b186 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '455d451ce5fb496e9fffe22a1ca00a41'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:41:47.052102 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-2c2578b9-d9bb-411f-8c5d-aee3f292b186 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '455d451ce5fb496e9fffe22a1ca00a41'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:41:47.061135 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1233/2440] 10.222.0.50 () {58 vars in 1224 bytes} [Tue Sep 13 16:41:46 2022] DELETE /identity/v3/users/455d451ce5fb496e9fffe22a1ca00a41 => generated 0 bytes in 234 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 13 16:41:47.071224 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.auth.core [None req-cad8fb1f-4a21-402d-8d91-029a29ccf6db None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81438) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:41:47.072494 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-cad8fb1f-4a21-402d-8d91-029a29ccf6db None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:41:47.079163 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4c4935a-f091-4e74-8e63-31778e9efab0 None None] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:41:47.080586 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4c4935a-f091-4e74-8e63-31778e9efab0 None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:41:47.081728 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4c4935a-f091-4e74-8e63-31778e9efab0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:41:47.117140 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.auth.core [None req-a4c4935a-f091-4e74-8e63-31778e9efab0 None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81437) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:41:47.119074 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-a4c4935a-f091-4e74-8e63-31778e9efab0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:41:47.128536 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=86,keystone:DELETE=12,keystone:INSERT=2 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:41:47.182309 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1208/2441] 10.222.0.50 () {58 vars in 958 bytes} [Tue Sep 13 16:41:47 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 161 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:41:47.184281 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f7ce8f9-eb83-401a-9f75-7f3ed9f45184 None None] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:41:47.184572 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f7ce8f9-eb83-401a-9f75-7f3ed9f45184 None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:41:47.184855 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f7ce8f9-eb83-401a-9f75-7f3ed9f45184 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:41:47.212273 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1234/2442] 10.222.0.50 () {58 vars in 958 bytes} [Tue Sep 13 16:41:47 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 146 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 13 16:41:47.219277 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.auth.core [None req-2f7ce8f9-eb83-401a-9f75-7f3ed9f45184 None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81438) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:41:47.222427 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-2f7ce8f9-eb83-401a-9f75-7f3ed9f45184 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:41:47.224064 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4c4935a-f091-4e74-8e63-31778e9efab0 None None] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:41:47.225493 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-a4c4935a-f091-4e74-8e63-31778e9efab0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:41:47.236704 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1209/2443] 10.222.0.50 () {58 vars in 958 bytes} [Tue Sep 13 16:41:47 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 53 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:41:47.237691 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f7ce8f9-eb83-401a-9f75-7f3ed9f45184 None None] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:41:47.251508 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f7ce8f9-eb83-401a-9f75-7f3ed9f45184 None None] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:41:47.257665 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3695d96-6505-41d2-aeac-9351b43b00d1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:41:47.258313 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3695d96-6505-41d2-aeac-9351b43b00d1 admin admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:41:47.258598 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3695d96-6505-41d2-aeac-9351b43b00d1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:41:47.259457 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3695d96-6505-41d2-aeac-9351b43b00d1 admin admin] PATH_INFO: `/v3/domains` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:41:47.260129 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3695d96-6505-41d2-aeac-9351b43b00d1 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81438) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 16:41:47.260713 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3695d96-6505-41d2-aeac-9351b43b00d1 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:41:47.262476 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3695d96-6505-41d2-aeac-9351b43b00d1 admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:41:47.279587 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1210/2444] 10.222.0.50 () {58 vars in 1153 bytes} [Tue Sep 13 16:41:47 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 42 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 16:41:47.280291 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3695d96-6505-41d2-aeac-9351b43b00d1 admin admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:41:47.294915 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3695d96-6505-41d2-aeac-9351b43b00d1 admin admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:41:47.301823 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72ba422c-e3dd-4212-9e50-76522a93f83f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:41:47.302716 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72ba422c-e3dd-4212-9e50-76522a93f83f admin admin] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:41:47.303207 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72ba422c-e3dd-4212-9e50-76522a93f83f admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:41:47.303671 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72ba422c-e3dd-4212-9e50-76522a93f83f admin admin] PATH_INFO: `/v3/projects` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:41:47.305098 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72ba422c-e3dd-4212-9e50-76522a93f83f admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:41:47.308139 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72ba422c-e3dd-4212-9e50-76522a93f83f admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:41:47.336831 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4c4935a-f091-4e74-8e63-31778e9efab0 None None] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:41:47.346593 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fa39a6b-6624-4c93-8cc1-ed60ced00bdc service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f436f07939d4855a21b195bc7566244', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:41:47.347661 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fa39a6b-6624-4c93-8cc1-ed60ced00bdc service neutron] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:41:47.348090 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fa39a6b-6624-4c93-8cc1-ed60ced00bdc service neutron] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:41:47.349051 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fa39a6b-6624-4c93-8cc1-ed60ced00bdc service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:41:47.365081 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fa39a6b-6624-4c93-8cc1-ed60ced00bdc service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:41:47.366995 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fa39a6b-6624-4c93-8cc1-ed60ced00bdc service neutron] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:41:47.370266 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1211/2445] 10.222.0.50 () {60 vars in 1153 bytes} [Tue Sep 13 16:41:47 2022] POST /identity/v3/projects => generated 379 bytes in 90 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 16:41:47.375083 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72ba422c-e3dd-4212-9e50-76522a93f83f admin admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:41:47.382076 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1235/2446] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 16:41:47 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 158 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 16:41:47.405958 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fa39a6b-6624-4c93-8cc1-ed60ced00bdc service neutron] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:41:47.408525 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-6fa39a6b-6624-4c93-8cc1-ed60ced00bdc service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:41:47.430878 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fa39a6b-6624-4c93-8cc1-ed60ced00bdc service neutron] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:41:47.465730 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72ba422c-e3dd-4212-9e50-76522a93f83f admin admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:41:47.475880 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6900fb6-a845-400c-8dc3-46c5924ee71d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:41:47.476986 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6900fb6-a845-400c-8dc3-46c5924ee71d admin admin] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:41:47.477429 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6900fb6-a845-400c-8dc3-46c5924ee71d admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:41:47.477973 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6900fb6-a845-400c-8dc3-46c5924ee71d admin admin] PATH_INFO: `/v3/users` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:41:47.479200 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6900fb6-a845-400c-8dc3-46c5924ee71d admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:41:47.481662 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6900fb6-a845-400c-8dc3-46c5924ee71d admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:41:47.511510 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1212/2447] 10.222.0.50 () {60 vars in 1144 bytes} [Tue Sep 13 16:41:47 2022] POST /identity/v3/users => generated 290 bytes in 138 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 16:41:47.517404 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6900fb6-a845-400c-8dc3-46c5924ee71d admin admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:41:47.540212 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6900fb6-a845-400c-8dc3-46c5924ee71d admin admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:41:47.557266 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0f00cc5-702e-4177-b399-243963492b05 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:41:47.558817 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0f00cc5-702e-4177-b399-243963492b05 admin admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:41:47.559328 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0f00cc5-702e-4177-b399-243963492b05 admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:41:47.559818 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0f00cc5-702e-4177-b399-243963492b05 admin admin] PATH_INFO: `/v3/roles` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:41:47.560674 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0f00cc5-702e-4177-b399-243963492b05 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:41:47.565026 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0f00cc5-702e-4177-b399-243963492b05 admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:41:47.569985 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=94,keystone:DELETE=5,keystone:INSERT=1 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:41:47.588572 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e0ad5c5-8368-4b17-80f6-8802a89ef27a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:41:47.589379 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e0ad5c5-8368-4b17-80f6-8802a89ef27a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:41:47.589759 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e0ad5c5-8368-4b17-80f6-8802a89ef27a admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:41:47.590104 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e0ad5c5-8368-4b17-80f6-8802a89ef27a admin admin] PATH_INFO: `/v3/users/61b2fd10440e4267a71553f6d6f793a8` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:41:47.592185 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e0ad5c5-8368-4b17-80f6-8802a89ef27a admin admin] RBAC: Authorizing `identity:delete_user(user_id=61b2fd10440e4267a71553f6d6f793a8)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:41:47.595725 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1213/2448] 10.222.0.50 () {58 vars in 1122 bytes} [Tue Sep 13 16:41:47 2022] GET /identity/v3/roles => generated 1420 bytes in 80 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 16:41:47.599718 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0f00cc5-702e-4177-b399-243963492b05 admin admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:41:47.599963 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e0ad5c5-8368-4b17-80f6-8802a89ef27a admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:41:47.618494 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0f00cc5-702e-4177-b399-243963492b05 admin admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:41:47.636025 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8b1bb10-68da-4c92-9523-45610f5b9aa0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:41:47.636662 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8b1bb10-68da-4c92-9523-45610f5b9aa0 admin admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:41:47.636945 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8b1bb10-68da-4c92-9523-45610f5b9aa0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:41:47.637222 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8b1bb10-68da-4c92-9523-45610f5b9aa0 admin admin] PATH_INFO: `/v3/roles` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:41:47.637871 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8b1bb10-68da-4c92-9523-45610f5b9aa0 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:41:47.644542 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8b1bb10-68da-4c92-9523-45610f5b9aa0 admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:41:47.676313 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1214/2449] 10.222.0.50 () {58 vars in 1122 bytes} [Tue Sep 13 16:41:47 2022] GET /identity/v3/roles => generated 1420 bytes in 74 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 16:41:47.677773 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8b1bb10-68da-4c92-9523-45610f5b9aa0 admin admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:41:47.685512 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-1e0ad5c5-8368-4b17-80f6-8802a89ef27a admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '61b2fd10440e4267a71553f6d6f793a8'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:41:47.699002 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8b1bb10-68da-4c92-9523-45610f5b9aa0 admin admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:41:47.703591 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-1e0ad5c5-8368-4b17-80f6-8802a89ef27a admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '61b2fd10440e4267a71553f6d6f793a8'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:41:47.715220 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56853b59-2b40-4c1c-9fc4-f2376e8ff058 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:41:47.715918 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56853b59-2b40-4c1c-9fc4-f2376e8ff058 admin admin] REQUEST_METHOD: `PUT` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:41:47.716198 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56853b59-2b40-4c1c-9fc4-f2376e8ff058 admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:41:47.716469 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56853b59-2b40-4c1c-9fc4-f2376e8ff058 admin admin] PATH_INFO: `/v3/projects/463a112433c04cde96b50aa32d5d01e4/users/78c03dc3584e4918bccf6f08ab00f331/roles/15db4aaca3264ae29bde38903ba43471` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:41:47.730263 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-1e0ad5c5-8368-4b17-80f6-8802a89ef27a admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '61b2fd10440e4267a71553f6d6f793a8'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:41:47.731536 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-1e0ad5c5-8368-4b17-80f6-8802a89ef27a admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '61b2fd10440e4267a71553f6d6f793a8'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:41:47.743525 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-1e0ad5c5-8368-4b17-80f6-8802a89ef27a admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '61b2fd10440e4267a71553f6d6f793a8'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:41:47.744023 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56853b59-2b40-4c1c-9fc4-f2376e8ff058 admin admin] RBAC: Authorizing `identity:create_grant(project_id=463a112433c04cde96b50aa32d5d01e4, user_id=78c03dc3584e4918bccf6f08ab00f331, role_id=15db4aaca3264ae29bde38903ba43471)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:41:47.747314 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56853b59-2b40-4c1c-9fc4-f2376e8ff058 admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:41:47.753013 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1236/2450] 10.222.0.50 () {58 vars in 1224 bytes} [Tue Sep 13 16:41:47 2022] DELETE /identity/v3/users/61b2fd10440e4267a71553f6d6f793a8 => generated 0 bytes in 348 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 16:41:47.757692 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7add9f5-a362-45a5-b2e8-761f78b2c865 None None] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:41:47.758097 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7add9f5-a362-45a5-b2e8-761f78b2c865 None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:41:47.758440 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7add9f5-a362-45a5-b2e8-761f78b2c865 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:41:47.784179 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1215/2451] 10.222.0.50 () {60 vars in 1483 bytes} [Tue Sep 13 16:41:47 2022] PUT /identity/v3/projects/463a112433c04cde96b50aa32d5d01e4/users/78c03dc3584e4918bccf6f08ab00f331/roles/15db4aaca3264ae29bde38903ba43471 => generated 0 bytes in 98 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 16:41:47.786763 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-365cd992-945d-4f89-b46d-a0b824146215 None None] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:41:47.787142 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-365cd992-945d-4f89-b46d-a0b824146215 None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:41:47.787476 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-365cd992-945d-4f89-b46d-a0b824146215 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:41:47.795680 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.auth.core [None req-a7add9f5-a362-45a5-b2e8-761f78b2c865 None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81437) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:41:47.799172 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-a7add9f5-a362-45a5-b2e8-761f78b2c865 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:41:47.844444 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.auth.core [None req-365cd992-945d-4f89-b46d-a0b824146215 None None] MFA Rules not processed for user `78c03dc3584e4918bccf6f08ab00f331`. Rule list: `[]` (Enabled: `True`). {{(pid=81438) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:41:47.846564 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-365cd992-945d-4f89-b46d-a0b824146215 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:41:47.889865 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:41:47.901366 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1237/2452] 10.222.0.50 () {58 vars in 958 bytes} [Tue Sep 13 16:41:47 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 145 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:41:47.916728 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7add9f5-a362-45a5-b2e8-761f78b2c865 None None] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:41:47.918486 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-a7add9f5-a362-45a5-b2e8-761f78b2c865 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:41:47.974505 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1216/2453] 10.222.0.50 () {58 vars in 958 bytes} [Tue Sep 13 16:41:47 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 189 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 13 16:41:47.979974 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-365cd992-945d-4f89-b46d-a0b824146215 None None] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:41:47.981530 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-365cd992-945d-4f89-b46d-a0b824146215 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:41:48.015994 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-365cd992-945d-4f89-b46d-a0b824146215 None None] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:41:48.031440 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cff59c69-cd5d-4cb5-9655-9c08320f2bc3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:41:48.032081 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cff59c69-cd5d-4cb5-9655-9c08320f2bc3 admin admin] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:41:48.032367 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cff59c69-cd5d-4cb5-9655-9c08320f2bc3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:41:48.032679 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cff59c69-cd5d-4cb5-9655-9c08320f2bc3 admin admin] PATH_INFO: `/v3/projects` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:41:48.033337 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cff59c69-cd5d-4cb5-9655-9c08320f2bc3 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:41:48.037173 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cff59c69-cd5d-4cb5-9655-9c08320f2bc3 admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:41:48.115736 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7add9f5-a362-45a5-b2e8-761f78b2c865 None None] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:41:48.122530 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1374114-d42a-4b06-97a1-a4b96351c709 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f436f07939d4855a21b195bc7566244', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:41:48.123523 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1374114-d42a-4b06-97a1-a4b96351c709 service neutron] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:41:48.123883 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1374114-d42a-4b06-97a1-a4b96351c709 service neutron] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:41:48.124487 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1374114-d42a-4b06-97a1-a4b96351c709 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:41:48.155839 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1374114-d42a-4b06-97a1-a4b96351c709 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:41:48.163978 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1374114-d42a-4b06-97a1-a4b96351c709 service neutron] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:41:48.164181 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1217/2454] 10.222.0.50 () {60 vars in 1153 bytes} [Tue Sep 13 16:41:47 2022] POST /identity/v3/projects => generated 381 bytes in 183 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 16:41:48.165400 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cff59c69-cd5d-4cb5-9655-9c08320f2bc3 admin admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:41:48.167639 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-cff59c69-cd5d-4cb5-9655-9c08320f2bc3 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:41:48.211116 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1238/2455] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 16:41:47 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 295 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 13 16:41:48.212363 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1374114-d42a-4b06-97a1-a4b96351c709 service neutron] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:41:48.238854 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1374114-d42a-4b06-97a1-a4b96351c709 service neutron] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:41:48.275955 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:41:48.301843 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cff59c69-cd5d-4cb5-9655-9c08320f2bc3 admin admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:41:48.309311 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4860c41c-a1ba-4430-9ea0-493cba8b13fd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:41:48.309746 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4860c41c-a1ba-4430-9ea0-493cba8b13fd admin admin] REQUEST_METHOD: `DELETE` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:41:48.310037 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4860c41c-a1ba-4430-9ea0-493cba8b13fd admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:41:48.310315 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4860c41c-a1ba-4430-9ea0-493cba8b13fd admin admin] PATH_INFO: `/v3/projects/05b6ba0827e348c492e84e80a00edd4c` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:41:48.311778 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4860c41c-a1ba-4430-9ea0-493cba8b13fd admin admin] RBAC: Authorizing `identity:delete_project(project_id=05b6ba0827e348c492e84e80a00edd4c)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:41:48.314726 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4860c41c-a1ba-4430-9ea0-493cba8b13fd admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:41:48.396423 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccdf5f75-3640-4c47-a842-fd0956bd25de admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:41:48.397058 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccdf5f75-3640-4c47-a842-fd0956bd25de admin admin] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:41:48.397343 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccdf5f75-3640-4c47-a842-fd0956bd25de admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:41:48.399819 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccdf5f75-3640-4c47-a842-fd0956bd25de admin admin] PATH_INFO: `/v3/users` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:41:48.401037 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ccdf5f75-3640-4c47-a842-fd0956bd25de admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:41:48.405296 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ccdf5f75-3640-4c47-a842-fd0956bd25de admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:41:48.408964 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-4860c41c-a1ba-4430-9ea0-493cba8b13fd admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '05b6ba0827e348c492e84e80a00edd4c', 'request_id': 'req-4860c41c-a1ba-4430-9ea0-493cba8b13fd'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:41:48.429094 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-4860c41c-a1ba-4430-9ea0-493cba8b13fd admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '05b6ba0827e348c492e84e80a00edd4c', 'request_id': 'req-4860c41c-a1ba-4430-9ea0-493cba8b13fd'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:41:48.445229 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-4860c41c-a1ba-4430-9ea0-493cba8b13fd admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '05b6ba0827e348c492e84e80a00edd4c', 'request_id': 'req-4860c41c-a1ba-4430-9ea0-493cba8b13fd'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:41:48.452803 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1239/2456] 10.222.0.50 () {60 vars in 1144 bytes} [Tue Sep 13 16:41:48 2022] POST /identity/v3/users => generated 297 bytes in 241 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 16:41:48.457286 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccdf5f75-3640-4c47-a842-fd0956bd25de admin admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:41:48.474137 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-4860c41c-a1ba-4430-9ea0-493cba8b13fd admin admin] The token cache is being invalidate because project 05b6ba0827e348c492e84e80a00edd4c was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81438) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 13 16:41:48.475636 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-4860c41c-a1ba-4430-9ea0-493cba8b13fd admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:41:48.478782 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-4860c41c-a1ba-4430-9ea0-493cba8b13fd admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:41:48.480604 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1218/2457] 10.222.0.50 () {58 vars in 1233 bytes} [Tue Sep 13 16:41:48 2022] DELETE /identity/v3/projects/05b6ba0827e348c492e84e80a00edd4c => generated 0 bytes in 318 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 16:41:48.481534 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccdf5f75-3640-4c47-a842-fd0956bd25de admin admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:41:48.484784 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4860c41c-a1ba-4430-9ea0-493cba8b13fd admin admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:41:48.489743 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-4860c41c-a1ba-4430-9ea0-493cba8b13fd admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:41:48.596435 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=91,keystone:INSERT=2,keystone:DELETE=7 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:41:48.622001 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3924f9d-ad29-4acc-9c05-7b6104cb5109 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:41:48.623376 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3924f9d-ad29-4acc-9c05-7b6104cb5109 admin admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:41:48.624140 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3924f9d-ad29-4acc-9c05-7b6104cb5109 admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:41:48.625259 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3924f9d-ad29-4acc-9c05-7b6104cb5109 admin admin] PATH_INFO: `/v3/roles` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:41:48.625371 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4860c41c-a1ba-4430-9ea0-493cba8b13fd admin admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:41:48.626637 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3924f9d-ad29-4acc-9c05-7b6104cb5109 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:41:48.632381 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3924f9d-ad29-4acc-9c05-7b6104cb5109 admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:41:48.640022 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d89b5ee5-dae8-49fb-a6c9-7b69e7441a17 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:41:48.640924 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d89b5ee5-dae8-49fb-a6c9-7b69e7441a17 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:41:48.641246 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d89b5ee5-dae8-49fb-a6c9-7b69e7441a17 admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:41:48.641725 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d89b5ee5-dae8-49fb-a6c9-7b69e7441a17 admin admin] PATH_INFO: `/v3/users/f464a6c33ea24c7380ba14041b233901` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:41:48.644367 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d89b5ee5-dae8-49fb-a6c9-7b69e7441a17 admin admin] RBAC: Authorizing `identity:delete_user(user_id=f464a6c33ea24c7380ba14041b233901)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:41:48.649282 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d89b5ee5-dae8-49fb-a6c9-7b69e7441a17 admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:41:48.657435 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1240/2458] 10.222.0.50 () {58 vars in 1122 bytes} [Tue Sep 13 16:41:48 2022] GET /identity/v3/roles => generated 1420 bytes in 201 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 16:41:48.664881 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3924f9d-ad29-4acc-9c05-7b6104cb5109 admin admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:41:48.664881 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-f3924f9d-ad29-4acc-9c05-7b6104cb5109 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:41:48.680097 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3924f9d-ad29-4acc-9c05-7b6104cb5109 admin admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:41:48.687537 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e676bcd-300d-4b31-9800-c4b70f2ebdcf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:41:48.687900 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e676bcd-300d-4b31-9800-c4b70f2ebdcf admin admin] REQUEST_METHOD: `PUT` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:41:48.688335 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e676bcd-300d-4b31-9800-c4b70f2ebdcf admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:41:48.688712 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e676bcd-300d-4b31-9800-c4b70f2ebdcf admin admin] PATH_INFO: `/v3/projects/dc73199ebe994b159e5877db5b604b3e/users/0cd79a586b2f407b8ae6b19266c67036/roles/031d4e1a23e34f71858fc4a6dfbd4ccb` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:41:48.734457 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e676bcd-300d-4b31-9800-c4b70f2ebdcf admin admin] RBAC: Authorizing `identity:create_grant(project_id=dc73199ebe994b159e5877db5b604b3e, user_id=0cd79a586b2f407b8ae6b19266c67036, role_id=031d4e1a23e34f71858fc4a6dfbd4ccb)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:41:48.742684 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e676bcd-300d-4b31-9800-c4b70f2ebdcf admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:41:48.759840 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-d89b5ee5-dae8-49fb-a6c9-7b69e7441a17 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'f464a6c33ea24c7380ba14041b233901'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:41:48.770937 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-d89b5ee5-dae8-49fb-a6c9-7b69e7441a17 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'f464a6c33ea24c7380ba14041b233901'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:41:48.772286 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1241/2459] 10.222.0.50 () {60 vars in 1483 bytes} [Tue Sep 13 16:41:48 2022] PUT /identity/v3/projects/dc73199ebe994b159e5877db5b604b3e/users/0cd79a586b2f407b8ae6b19266c67036/roles/031d4e1a23e34f71858fc4a6dfbd4ccb => generated 0 bytes in 111 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 13 16:41:48.773281 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e676bcd-300d-4b31-9800-c4b70f2ebdcf admin admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:41:48.776820 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-0e676bcd-300d-4b31-9800-c4b70f2ebdcf admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:41:48.789003 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-d89b5ee5-dae8-49fb-a6c9-7b69e7441a17 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'f464a6c33ea24c7380ba14041b233901'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:41:48.790117 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-d89b5ee5-dae8-49fb-a6c9-7b69e7441a17 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'f464a6c33ea24c7380ba14041b233901'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:41:48.808785 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-d89b5ee5-dae8-49fb-a6c9-7b69e7441a17 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'f464a6c33ea24c7380ba14041b233901'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:41:48.816039 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1219/2460] 10.222.0.50 () {58 vars in 1224 bytes} [Tue Sep 13 16:41:48 2022] DELETE /identity/v3/users/f464a6c33ea24c7380ba14041b233901 => generated 0 bytes in 332 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 13 16:41:48.817459 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d89b5ee5-dae8-49fb-a6c9-7b69e7441a17 admin admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:41:48.819030 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-d89b5ee5-dae8-49fb-a6c9-7b69e7441a17 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:41:48.887724 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e676bcd-300d-4b31-9800-c4b70f2ebdcf admin admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:41:48.949121 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:41:48.990021 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d89b5ee5-dae8-49fb-a6c9-7b69e7441a17 admin admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:41:49.002740 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbca03ed-fe89-4c65-be2b-7bb7127a8cee admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:41:49.004858 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbca03ed-fe89-4c65-be2b-7bb7127a8cee admin admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:41:49.005162 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbca03ed-fe89-4c65-be2b-7bb7127a8cee admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:41:49.006443 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbca03ed-fe89-4c65-be2b-7bb7127a8cee admin admin] PATH_INFO: `/v3/roles` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:41:49.006443 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbca03ed-fe89-4c65-be2b-7bb7127a8cee admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:41:49.008846 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbca03ed-fe89-4c65-be2b-7bb7127a8cee admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:41:49.029435 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-584475c2-7d6a-435d-81ef-9e8536c052f0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:41:49.030925 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1220/2461] 10.222.0.50 () {58 vars in 1122 bytes} [Tue Sep 13 16:41:48 2022] GET /identity/v3/roles => generated 1420 bytes in 214 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 16:41:49.031590 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-584475c2-7d6a-435d-81ef-9e8536c052f0 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:41:49.031844 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbca03ed-fe89-4c65-be2b-7bb7127a8cee admin admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:41:49.031944 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-584475c2-7d6a-435d-81ef-9e8536c052f0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:41:49.032269 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-584475c2-7d6a-435d-81ef-9e8536c052f0 admin admin] PATH_INFO: `/v3/projects/e4a3fac5db2443b492844028e5cbee61` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:41:49.035471 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-584475c2-7d6a-435d-81ef-9e8536c052f0 admin admin] RBAC: Authorizing `identity:delete_project(project_id=e4a3fac5db2443b492844028e5cbee61)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:41:49.039816 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-584475c2-7d6a-435d-81ef-9e8536c052f0 admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:41:49.053214 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbca03ed-fe89-4c65-be2b-7bb7127a8cee admin admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:41:49.067494 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d0270fc-cab2-421e-b161-8c2e2e6af73e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:41:49.068192 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d0270fc-cab2-421e-b161-8c2e2e6af73e admin admin] REQUEST_METHOD: `PUT` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:41:49.068480 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d0270fc-cab2-421e-b161-8c2e2e6af73e admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:41:49.068767 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d0270fc-cab2-421e-b161-8c2e2e6af73e admin admin] PATH_INFO: `/v3/projects/dc73199ebe994b159e5877db5b604b3e/users/0cd79a586b2f407b8ae6b19266c67036/roles/031d4e1a23e34f71858fc4a6dfbd4ccb` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:41:49.076141 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d0270fc-cab2-421e-b161-8c2e2e6af73e admin admin] RBAC: Authorizing `identity:create_grant(project_id=dc73199ebe994b159e5877db5b604b3e, user_id=0cd79a586b2f407b8ae6b19266c67036, role_id=031d4e1a23e34f71858fc4a6dfbd4ccb)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:41:49.078806 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d0270fc-cab2-421e-b161-8c2e2e6af73e admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:41:49.106951 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1221/2462] 10.222.0.50 () {60 vars in 1483 bytes} [Tue Sep 13 16:41:49 2022] PUT /identity/v3/projects/dc73199ebe994b159e5877db5b604b3e/users/0cd79a586b2f407b8ae6b19266c67036/roles/031d4e1a23e34f71858fc4a6dfbd4ccb => generated 0 bytes in 75 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 16:41:49.121384 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ce92a67-da06-497c-a802-727741724e97 None None] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:41:49.121733 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ce92a67-da06-497c-a802-727741724e97 None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:41:49.122019 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ce92a67-da06-497c-a802-727741724e97 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:41:49.156271 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-584475c2-7d6a-435d-81ef-9e8536c052f0 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'e4a3fac5db2443b492844028e5cbee61', 'request_id': 'req-584475c2-7d6a-435d-81ef-9e8536c052f0'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:41:49.172487 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-584475c2-7d6a-435d-81ef-9e8536c052f0 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'e4a3fac5db2443b492844028e5cbee61', 'request_id': 'req-584475c2-7d6a-435d-81ef-9e8536c052f0'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:41:49.182848 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.auth.core [None req-8ce92a67-da06-497c-a802-727741724e97 None None] MFA Rules not processed for user `0cd79a586b2f407b8ae6b19266c67036`. Rule list: `[]` (Enabled: `True`). {{(pid=81438) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:41:49.185375 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-8ce92a67-da06-497c-a802-727741724e97 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:41:49.187929 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-584475c2-7d6a-435d-81ef-9e8536c052f0 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'e4a3fac5db2443b492844028e5cbee61', 'request_id': 'req-584475c2-7d6a-435d-81ef-9e8536c052f0'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:41:49.209851 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-584475c2-7d6a-435d-81ef-9e8536c052f0 admin admin] The token cache is being invalidate because project e4a3fac5db2443b492844028e5cbee61 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81437) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 13 16:41:49.211019 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-584475c2-7d6a-435d-81ef-9e8536c052f0 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:41:49.214929 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-584475c2-7d6a-435d-81ef-9e8536c052f0 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:41:49.220076 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1242/2463] 10.222.0.50 () {58 vars in 1233 bytes} [Tue Sep 13 16:41:48 2022] DELETE /identity/v3/projects/e4a3fac5db2443b492844028e5cbee61 => generated 0 bytes in 444 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 16:41:49.221270 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-584475c2-7d6a-435d-81ef-9e8536c052f0 admin admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:41:49.226475 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-584475c2-7d6a-435d-81ef-9e8536c052f0 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:41:49.274755 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=88,keystone:DELETE=10,keystone:INSERT=2 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:41:49.386097 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1222/2464] 10.222.0.50 () {58 vars in 958 bytes} [Tue Sep 13 16:41:49 2022] POST /identity/v3/auth/tokens => generated 3325 bytes in 259 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:41:49.386494 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-584475c2-7d6a-435d-81ef-9e8536c052f0 admin admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:41:49.390697 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ce92a67-da06-497c-a802-727741724e97 None None] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:41:49.398529 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa09d6e3-045a-463b-b64f-c01f01e3eaf0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:41:49.408571 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa09d6e3-045a-463b-b64f-c01f01e3eaf0 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:41:49.408571 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa09d6e3-045a-463b-b64f-c01f01e3eaf0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:41:49.408962 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-8ce92a67-da06-497c-a802-727741724e97 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:41:49.409054 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa09d6e3-045a-463b-b64f-c01f01e3eaf0 admin admin] PATH_INFO: `/v3/users/de5c6be59405407b989a0e2b3ff08046` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:41:49.409054 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa09d6e3-045a-463b-b64f-c01f01e3eaf0 admin admin] RBAC: Authorizing `identity:delete_user(user_id=de5c6be59405407b989a0e2b3ff08046)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:41:49.409054 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa09d6e3-045a-463b-b64f-c01f01e3eaf0 admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:41:49.418998 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ce92a67-da06-497c-a802-727741724e97 None None] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:41:49.424486 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-696373d6-8dc5-47b7-991a-f0116fd90c2f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:41:49.425284 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-696373d6-8dc5-47b7-991a-f0116fd90c2f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:41:49.425676 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-696373d6-8dc5-47b7-991a-f0116fd90c2f admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:41:49.425973 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-696373d6-8dc5-47b7-991a-f0116fd90c2f admin admin] PATH_INFO: `/v3/projects/4bbbc6e825974a71b82562c8544dfa47` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:41:49.427281 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-696373d6-8dc5-47b7-991a-f0116fd90c2f admin admin] RBAC: Authorizing `identity:delete_project(project_id=4bbbc6e825974a71b82562c8544dfa47)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:41:49.429381 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-696373d6-8dc5-47b7-991a-f0116fd90c2f admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:41:49.471221 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-aa09d6e3-045a-463b-b64f-c01f01e3eaf0 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'de5c6be59405407b989a0e2b3ff08046'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:41:49.481450 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-aa09d6e3-045a-463b-b64f-c01f01e3eaf0 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'de5c6be59405407b989a0e2b3ff08046'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:41:49.487476 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=83,keystone:DELETE=15,keystone:INSERT=2 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:41:49.492290 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-696373d6-8dc5-47b7-991a-f0116fd90c2f admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '4bbbc6e825974a71b82562c8544dfa47', 'request_id': 'req-696373d6-8dc5-47b7-991a-f0116fd90c2f'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:41:49.501403 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-aa09d6e3-045a-463b-b64f-c01f01e3eaf0 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'de5c6be59405407b989a0e2b3ff08046'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:41:49.503203 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-696373d6-8dc5-47b7-991a-f0116fd90c2f admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '4bbbc6e825974a71b82562c8544dfa47', 'request_id': 'req-696373d6-8dc5-47b7-991a-f0116fd90c2f'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:41:49.503336 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-aa09d6e3-045a-463b-b64f-c01f01e3eaf0 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'de5c6be59405407b989a0e2b3ff08046'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:41:49.510472 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-696373d6-8dc5-47b7-991a-f0116fd90c2f admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '4bbbc6e825974a71b82562c8544dfa47', 'request_id': 'req-696373d6-8dc5-47b7-991a-f0116fd90c2f'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:41:49.512198 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-aa09d6e3-045a-463b-b64f-c01f01e3eaf0 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'de5c6be59405407b989a0e2b3ff08046'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:41:49.520149 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1243/2465] 10.222.0.50 () {58 vars in 1224 bytes} [Tue Sep 13 16:41:49 2022] DELETE /identity/v3/users/de5c6be59405407b989a0e2b3ff08046 => generated 0 bytes in 299 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 16:41:49.521162 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa09d6e3-045a-463b-b64f-c01f01e3eaf0 admin admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:41:49.522557 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-aa09d6e3-045a-463b-b64f-c01f01e3eaf0 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:41:49.526335 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-696373d6-8dc5-47b7-991a-f0116fd90c2f admin admin] The token cache is being invalidate because project 4bbbc6e825974a71b82562c8544dfa47 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81438) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 13 16:41:49.526624 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-696373d6-8dc5-47b7-991a-f0116fd90c2f admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:41:49.527410 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-696373d6-8dc5-47b7-991a-f0116fd90c2f admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:41:49.529252 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1223/2466] 10.222.0.50 () {58 vars in 1233 bytes} [Tue Sep 13 16:41:49 2022] DELETE /identity/v3/projects/4bbbc6e825974a71b82562c8544dfa47 => generated 0 bytes in 142 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 16:41:49.533609 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6e75c2a-6ef8-43d7-8a55-550f7db53342 None None] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:41:49.534060 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6e75c2a-6ef8-43d7-8a55-550f7db53342 None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:41:49.534806 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6e75c2a-6ef8-43d7-8a55-550f7db53342 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:41:49.568831 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.auth.core [None req-c6e75c2a-6ef8-43d7-8a55-550f7db53342 None None] MFA Rules not processed for user `b6417e189b0d467480627cfd38331a9f`. Rule list: `[]` (Enabled: `True`). {{(pid=81438) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:41:49.570067 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-c6e75c2a-6ef8-43d7-8a55-550f7db53342 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:41:49.628110 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa09d6e3-045a-463b-b64f-c01f01e3eaf0 admin admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:41:49.632219 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2efb57ce-9841-441e-81e8-22c7810b077a service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7fd6b5afd67b48f9b0f0597ccd4cf4df', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:41:49.632850 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2efb57ce-9841-441e-81e8-22c7810b077a service cinder] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:41:49.633139 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2efb57ce-9841-441e-81e8-22c7810b077a service cinder] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:41:49.633417 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2efb57ce-9841-441e-81e8-22c7810b077a service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:41:49.635409 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-2efb57ce-9841-441e-81e8-22c7810b077a service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:41:49.659217 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1224/2467] 10.222.0.50 () {58 vars in 958 bytes} [Tue Sep 13 16:41:49 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 130 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:41:49.660994 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-240a24f0-fbab-4588-9122-93e08da68eec None None] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:41:49.661180 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-240a24f0-fbab-4588-9122-93e08da68eec None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:41:49.661490 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-240a24f0-fbab-4588-9122-93e08da68eec None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:41:49.686117 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.auth.core [None req-240a24f0-fbab-4588-9122-93e08da68eec None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81438) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:41:49.687620 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-240a24f0-fbab-4588-9122-93e08da68eec None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:41:49.725011 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2efb57ce-9841-441e-81e8-22c7810b077a service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:41:49.726588 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2efb57ce-9841-441e-81e8-22c7810b077a service cinder] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:41:49.740946 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1244/2468] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 16:41:49 2022] GET /identity/v3/auth/tokens => generated 3325 bytes in 220 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 16:41:49.741536 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2efb57ce-9841-441e-81e8-22c7810b077a service cinder] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:41:49.743059 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-2efb57ce-9841-441e-81e8-22c7810b077a service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:41:49.765629 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:41:49.825109 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1225/2469] 10.222.0.50 () {58 vars in 958 bytes} [Tue Sep 13 16:41:49 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 165 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:41:49.833359 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-397f6294-c27b-43bb-93e1-f18f7cda9a40 None None] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:41:49.833659 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-397f6294-c27b-43bb-93e1-f18f7cda9a40 None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:41:49.833947 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-397f6294-c27b-43bb-93e1-f18f7cda9a40 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:41:49.869717 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.auth.core [None req-397f6294-c27b-43bb-93e1-f18f7cda9a40 None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81438) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:41:49.871155 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-397f6294-c27b-43bb-93e1-f18f7cda9a40 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:41:49.886006 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1226/2470] 10.222.0.50 () {58 vars in 958 bytes} [Tue Sep 13 16:41:49 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 53 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 13 16:41:49.887496 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-397f6294-c27b-43bb-93e1-f18f7cda9a40 None None] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:41:49.891094 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-397f6294-c27b-43bb-93e1-f18f7cda9a40 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:41:49.904429 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2efb57ce-9841-441e-81e8-22c7810b077a service cinder] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:41:49.910437 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-397f6294-c27b-43bb-93e1-f18f7cda9a40 None None] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:41:49.912004 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17724558-dcf4-4267-acba-ec4b6ec93fdc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:41:49.912665 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17724558-dcf4-4267-acba-ec4b6ec93fdc admin admin] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:41:49.912947 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17724558-dcf4-4267-acba-ec4b6ec93fdc admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:41:49.913407 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17724558-dcf4-4267-acba-ec4b6ec93fdc admin admin] PATH_INFO: `/v3/projects` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:41:49.914058 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17724558-dcf4-4267-acba-ec4b6ec93fdc admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:41:49.915409 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd6ba0bb-e78e-4079-a98a-b01cbab1efa6 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7fd6b5afd67b48f9b0f0597ccd4cf4df', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:41:49.916016 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd6ba0bb-e78e-4079-a98a-b01cbab1efa6 service cinder] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:41:49.916211 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17724558-dcf4-4267-acba-ec4b6ec93fdc admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:41:49.916305 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd6ba0bb-e78e-4079-a98a-b01cbab1efa6 service cinder] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:41:49.916553 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd6ba0bb-e78e-4079-a98a-b01cbab1efa6 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:41:49.918340 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-cd6ba0bb-e78e-4079-a98a-b01cbab1efa6 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:41:49.958445 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1245/2471] 10.222.0.50 () {60 vars in 1153 bytes} [Tue Sep 13 16:41:49 2022] POST /identity/v3/projects => generated 379 bytes in 217 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 16:41:49.962618 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17724558-dcf4-4267-acba-ec4b6ec93fdc admin admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:41:49.988730 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:DELETE=2,keystone:SELECT=97,keystone:INSERT=1 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:41:50.037555 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd6ba0bb-e78e-4079-a98a-b01cbab1efa6 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:41:50.039104 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd6ba0bb-e78e-4079-a98a-b01cbab1efa6 service cinder] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:41:50.053298 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17724558-dcf4-4267-acba-ec4b6ec93fdc admin admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:41:50.053999 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1227/2472] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 16:41:49 2022] GET /identity/v3/auth/tokens => generated 3256 bytes in 167 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 16:41:50.054959 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd6ba0bb-e78e-4079-a98a-b01cbab1efa6 service cinder] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:41:50.058185 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-037c29fb-af17-4bb7-9ecf-fc3558bc373c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:41:50.058799 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-037c29fb-af17-4bb7-9ecf-fc3558bc373c admin admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:41:50.059126 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-037c29fb-af17-4bb7-9ecf-fc3558bc373c admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:41:50.059399 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-037c29fb-af17-4bb7-9ecf-fc3558bc373c admin admin] PATH_INFO: `/v3/domains` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:41:50.060060 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-037c29fb-af17-4bb7-9ecf-fc3558bc373c admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81437) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 16:41:50.060364 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-037c29fb-af17-4bb7-9ecf-fc3558bc373c admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:41:50.062318 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-037c29fb-af17-4bb7-9ecf-fc3558bc373c admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:41:50.073860 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd6ba0bb-e78e-4079-a98a-b01cbab1efa6 service cinder] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:41:50.075895 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1246/2473] 10.222.0.50 () {58 vars in 1153 bytes} [Tue Sep 13 16:41:49 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 114 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 16:41:50.080274 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85e2d8a4-8fa3-46c9-966a-77e0dcd29cfc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:41:50.080958 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85e2d8a4-8fa3-46c9-966a-77e0dcd29cfc admin admin] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:41:50.081183 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85e2d8a4-8fa3-46c9-966a-77e0dcd29cfc admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:41:50.081287 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-037c29fb-af17-4bb7-9ecf-fc3558bc373c admin admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:41:50.081457 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85e2d8a4-8fa3-46c9-966a-77e0dcd29cfc admin admin] PATH_INFO: `/v3/users` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:41:50.082093 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85e2d8a4-8fa3-46c9-966a-77e0dcd29cfc admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:41:50.083928 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-037c29fb-af17-4bb7-9ecf-fc3558bc373c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:41:50.084123 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85e2d8a4-8fa3-46c9-966a-77e0dcd29cfc admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:41:50.098685 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-037c29fb-af17-4bb7-9ecf-fc3558bc373c admin admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:41:50.104554 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa433a59-c8bd-4b9e-83d8-9e6f795c9c34 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:41:50.105331 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa433a59-c8bd-4b9e-83d8-9e6f795c9c34 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:41:50.105606 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa433a59-c8bd-4b9e-83d8-9e6f795c9c34 admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:41:50.105895 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa433a59-c8bd-4b9e-83d8-9e6f795c9c34 admin admin] PATH_INFO: `/v3/projects/aa19ae24f9344604aa94f624eb110bc9` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:41:50.107084 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa433a59-c8bd-4b9e-83d8-9e6f795c9c34 admin admin] RBAC: Authorizing `identity:delete_project(project_id=aa19ae24f9344604aa94f624eb110bc9)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:41:50.109012 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa433a59-c8bd-4b9e-83d8-9e6f795c9c34 admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:41:50.114364 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1228/2474] 10.222.0.50 () {60 vars in 1144 bytes} [Tue Sep 13 16:41:50 2022] POST /identity/v3/users => generated 296 bytes in 57 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 16:41:50.114364 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85e2d8a4-8fa3-46c9-966a-77e0dcd29cfc admin admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:41:50.126056 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85e2d8a4-8fa3-46c9-966a-77e0dcd29cfc admin admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:41:50.131973 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eea46e62-598a-4d1b-8481-05b1baa95409 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:41:50.132641 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eea46e62-598a-4d1b-8481-05b1baa95409 admin admin] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:41:50.132967 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eea46e62-598a-4d1b-8481-05b1baa95409 admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:41:50.133239 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eea46e62-598a-4d1b-8481-05b1baa95409 admin admin] PATH_INFO: `/v3/projects` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:41:50.133880 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eea46e62-598a-4d1b-8481-05b1baa95409 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:41:50.137272 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eea46e62-598a-4d1b-8481-05b1baa95409 admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:41:50.181302 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1229/2475] 10.222.0.50 () {60 vars in 1153 bytes} [Tue Sep 13 16:41:50 2022] POST /identity/v3/projects => generated 387 bytes in 66 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 16:41:50.181302 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eea46e62-598a-4d1b-8481-05b1baa95409 admin admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:41:50.182411 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-fa433a59-c8bd-4b9e-83d8-9e6f795c9c34 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'aa19ae24f9344604aa94f624eb110bc9', 'request_id': 'req-fa433a59-c8bd-4b9e-83d8-9e6f795c9c34'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:41:50.192804 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-fa433a59-c8bd-4b9e-83d8-9e6f795c9c34 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'aa19ae24f9344604aa94f624eb110bc9', 'request_id': 'req-fa433a59-c8bd-4b9e-83d8-9e6f795c9c34'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:41:50.205788 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-fa433a59-c8bd-4b9e-83d8-9e6f795c9c34 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'aa19ae24f9344604aa94f624eb110bc9', 'request_id': 'req-fa433a59-c8bd-4b9e-83d8-9e6f795c9c34'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:41:50.230637 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-fa433a59-c8bd-4b9e-83d8-9e6f795c9c34 admin admin] The token cache is being invalidate because project aa19ae24f9344604aa94f624eb110bc9 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81437) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 13 16:41:50.230971 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-fa433a59-c8bd-4b9e-83d8-9e6f795c9c34 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:41:50.231956 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-fa433a59-c8bd-4b9e-83d8-9e6f795c9c34 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:41:50.233703 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1247/2476] 10.222.0.50 () {58 vars in 1233 bytes} [Tue Sep 13 16:41:50 2022] DELETE /identity/v3/projects/aa19ae24f9344604aa94f624eb110bc9 => generated 0 bytes in 154 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 16:41:50.234521 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa433a59-c8bd-4b9e-83d8-9e6f795c9c34 admin admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:41:50.236455 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-fa433a59-c8bd-4b9e-83d8-9e6f795c9c34 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:41:50.312931 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eea46e62-598a-4d1b-8481-05b1baa95409 admin admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:41:50.318842 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4dff2f8f-60d5-4d39-850d-c565a1f9dec0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:41:50.319659 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dff2f8f-60d5-4d39-850d-c565a1f9dec0 admin admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:41:50.320015 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dff2f8f-60d5-4d39-850d-c565a1f9dec0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:41:50.320443 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dff2f8f-60d5-4d39-850d-c565a1f9dec0 admin admin] PATH_INFO: `/v3/roles` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:41:50.323824 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4dff2f8f-60d5-4d39-850d-c565a1f9dec0 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:41:50.325596 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4dff2f8f-60d5-4d39-850d-c565a1f9dec0 admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:41:50.336651 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa433a59-c8bd-4b9e-83d8-9e6f795c9c34 admin admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:41:50.344648 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1230/2477] 10.222.0.50 () {58 vars in 1122 bytes} [Tue Sep 13 16:41:50 2022] GET /identity/v3/roles => generated 1420 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 16:41:50.344648 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1222244f-8715-42db-a7db-047642a9b046 None None] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:41:50.344648 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1222244f-8715-42db-a7db-047642a9b046 None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:41:50.344648 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1222244f-8715-42db-a7db-047642a9b046 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:41:50.346161 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd091a87-0ed8-405b-baa1-e420379a04a1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:41:50.346830 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd091a87-0ed8-405b-baa1-e420379a04a1 admin admin] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:41:50.347093 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd091a87-0ed8-405b-baa1-e420379a04a1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:41:50.347367 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd091a87-0ed8-405b-baa1-e420379a04a1 admin admin] PATH_INFO: `/v3/users` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:41:50.348048 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd091a87-0ed8-405b-baa1-e420379a04a1 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:41:50.350249 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd091a87-0ed8-405b-baa1-e420379a04a1 admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:41:50.389277 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.auth.core [None req-1222244f-8715-42db-a7db-047642a9b046 None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81438) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:41:50.396453 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1248/2478] 10.222.0.50 () {60 vars in 1144 bytes} [Tue Sep 13 16:41:50 2022] POST /identity/v3/users => generated 294 bytes in 159 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 16:41:50.396613 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-1222244f-8715-42db-a7db-047642a9b046 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:41:50.396694 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd091a87-0ed8-405b-baa1-e420379a04a1 admin admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:41:50.398769 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-fd091a87-0ed8-405b-baa1-e420379a04a1 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:41:50.403850 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1231/2479] 10.222.0.50 () {58 vars in 958 bytes} [Tue Sep 13 16:41:50 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 61 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:41:50.414336 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1222244f-8715-42db-a7db-047642a9b046 None None] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:41:50.418145 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd091a87-0ed8-405b-baa1-e420379a04a1 admin admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:41:50.425404 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1222244f-8715-42db-a7db-047642a9b046 None None] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:41:50.426090 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-840650b2-735f-4946-ad69-4d00f8ed7cda admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:41:50.426791 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-840650b2-735f-4946-ad69-4d00f8ed7cda admin admin] REQUEST_METHOD: `PUT` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:41:50.427103 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-840650b2-735f-4946-ad69-4d00f8ed7cda admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:41:50.427373 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-840650b2-735f-4946-ad69-4d00f8ed7cda admin admin] PATH_INFO: `/v3/projects/da3174247d754b528afbc974e8f9bc31/users/181b443239e8473b841ca01c13d1e446/roles/031d4e1a23e34f71858fc4a6dfbd4ccb` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:41:50.430625 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4924e3c-a3ac-4de6-8100-4c7e469ab2a5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:41:50.431907 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4924e3c-a3ac-4de6-8100-4c7e469ab2a5 admin admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:41:50.432471 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4924e3c-a3ac-4de6-8100-4c7e469ab2a5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:41:50.432471 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4924e3c-a3ac-4de6-8100-4c7e469ab2a5 admin admin] PATH_INFO: `/v3/roles` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:41:50.432870 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4924e3c-a3ac-4de6-8100-4c7e469ab2a5 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:41:50.435134 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4924e3c-a3ac-4de6-8100-4c7e469ab2a5 admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:41:50.446879 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-840650b2-735f-4946-ad69-4d00f8ed7cda admin admin] RBAC: Authorizing `identity:create_grant(project_id=da3174247d754b528afbc974e8f9bc31, user_id=181b443239e8473b841ca01c13d1e446, role_id=031d4e1a23e34f71858fc4a6dfbd4ccb)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:41:50.447244 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1232/2480] 10.222.0.50 () {58 vars in 1122 bytes} [Tue Sep 13 16:41:50 2022] GET /identity/v3/roles => generated 1420 bytes in 43 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 16:41:50.448586 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-661c322b-2170-4eee-9fc9-d1839fbe802d None None] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:41:50.448878 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-661c322b-2170-4eee-9fc9-d1839fbe802d None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:41:50.449156 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-661c322b-2170-4eee-9fc9-d1839fbe802d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:41:50.451286 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-840650b2-735f-4946-ad69-4d00f8ed7cda admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:41:50.454325 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=89,keystone:DELETE=7,keystone:INSERT=4 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:41:50.471419 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.auth.core [None req-661c322b-2170-4eee-9fc9-d1839fbe802d None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81438) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:41:50.472854 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-661c322b-2170-4eee-9fc9-d1839fbe802d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:41:50.475169 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1249/2481] 10.222.0.50 () {60 vars in 1483 bytes} [Tue Sep 13 16:41:50 2022] PUT /identity/v3/projects/da3174247d754b528afbc974e8f9bc31/users/181b443239e8473b841ca01c13d1e446/roles/031d4e1a23e34f71858fc4a6dfbd4ccb => generated 0 bytes in 78 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 16:41:50.476164 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-840650b2-735f-4946-ad69-4d00f8ed7cda admin admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:41:50.485697 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:41:50.551471 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-840650b2-735f-4946-ad69-4d00f8ed7cda admin admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:41:50.559086 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1233/2482] 10.222.0.50 () {58 vars in 958 bytes} [Tue Sep 13 16:41:50 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:41:50.559256 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a4428eb-bb02-4c4c-b7dd-1ca82ffaf60a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:41:50.559256 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a4428eb-bb02-4c4c-b7dd-1ca82ffaf60a admin admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:41:50.559465 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-661c322b-2170-4eee-9fc9-d1839fbe802d None None] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:41:50.559548 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a4428eb-bb02-4c4c-b7dd-1ca82ffaf60a admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:41:50.559548 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a4428eb-bb02-4c4c-b7dd-1ca82ffaf60a admin admin] PATH_INFO: `/v3/roles` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:41:50.559548 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a4428eb-bb02-4c4c-b7dd-1ca82ffaf60a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:41:50.560183 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a4428eb-bb02-4c4c-b7dd-1ca82ffaf60a admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:41:50.570359 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-661c322b-2170-4eee-9fc9-d1839fbe802d None None] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:41:50.571873 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1250/2483] 10.222.0.50 () {58 vars in 1122 bytes} [Tue Sep 13 16:41:50 2022] GET /identity/v3/roles => generated 1420 bytes in 96 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 16:41:50.572712 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a4428eb-bb02-4c4c-b7dd-1ca82ffaf60a admin admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:41:50.575765 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c6429ce-8e52-47fe-b52b-65ba6afc873f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:41:50.576386 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c6429ce-8e52-47fe-b52b-65ba6afc873f admin admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:41:50.576669 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c6429ce-8e52-47fe-b52b-65ba6afc873f admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:41:50.576945 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c6429ce-8e52-47fe-b52b-65ba6afc873f admin admin] PATH_INFO: `/v3/roles` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:41:50.577720 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c6429ce-8e52-47fe-b52b-65ba6afc873f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:41:50.579388 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c6429ce-8e52-47fe-b52b-65ba6afc873f admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:41:50.584518 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a4428eb-bb02-4c4c-b7dd-1ca82ffaf60a admin admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:41:50.589939 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2f84751-c70d-4b46-81c9-2d4980f01075 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:41:50.590549 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2f84751-c70d-4b46-81c9-2d4980f01075 admin admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:41:50.591084 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2f84751-c70d-4b46-81c9-2d4980f01075 admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:41:50.591188 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1234/2484] 10.222.0.50 () {58 vars in 1122 bytes} [Tue Sep 13 16:41:50 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 16:41:50.591790 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c6429ce-8e52-47fe-b52b-65ba6afc873f admin admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:41:50.594279 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2f84751-c70d-4b46-81c9-2d4980f01075 admin admin] PATH_INFO: `/v3/domains` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:41:50.595083 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2f84751-c70d-4b46-81c9-2d4980f01075 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81437) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 16:41:50.595564 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2f84751-c70d-4b46-81c9-2d4980f01075 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:41:50.597029 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2f84751-c70d-4b46-81c9-2d4980f01075 admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:41:50.602765 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c6429ce-8e52-47fe-b52b-65ba6afc873f admin admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:41:50.609414 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f022468f-3c12-483d-8c94-0ffec762926c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:41:50.610033 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f022468f-3c12-483d-8c94-0ffec762926c admin admin] REQUEST_METHOD: `PUT` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:41:50.610501 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1251/2485] 10.222.0.50 () {58 vars in 1153 bytes} [Tue Sep 13 16:41:50 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 16:41:50.610683 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f022468f-3c12-483d-8c94-0ffec762926c admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:41:50.610683 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f022468f-3c12-483d-8c94-0ffec762926c admin admin] PATH_INFO: `/v3/projects/c2310d76bc7c40d8ac8e1276ce60126f/users/d29755d7854441c8a5c85aa16c7a05f5/roles/15db4aaca3264ae29bde38903ba43471` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:41:50.611234 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2f84751-c70d-4b46-81c9-2d4980f01075 admin admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:41:50.621934 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2f84751-c70d-4b46-81c9-2d4980f01075 admin admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:41:50.628615 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f022468f-3c12-483d-8c94-0ffec762926c admin admin] RBAC: Authorizing `identity:create_grant(project_id=c2310d76bc7c40d8ac8e1276ce60126f, user_id=d29755d7854441c8a5c85aa16c7a05f5, role_id=15db4aaca3264ae29bde38903ba43471)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:41:50.629484 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adcd5695-005b-4fe6-baf1-2375c35f65dd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:41:50.630157 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adcd5695-005b-4fe6-baf1-2375c35f65dd admin admin] REQUEST_METHOD: `PUT` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:41:50.630435 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adcd5695-005b-4fe6-baf1-2375c35f65dd admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:41:50.630709 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adcd5695-005b-4fe6-baf1-2375c35f65dd admin admin] PATH_INFO: `/v3/projects/da3174247d754b528afbc974e8f9bc31/users/181b443239e8473b841ca01c13d1e446/roles/031d4e1a23e34f71858fc4a6dfbd4ccb` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:41:50.631597 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f022468f-3c12-483d-8c94-0ffec762926c admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:41:50.632730 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-adcd5695-005b-4fe6-baf1-2375c35f65dd admin admin] RBAC: Authorizing `identity:create_grant(project_id=da3174247d754b528afbc974e8f9bc31, user_id=181b443239e8473b841ca01c13d1e446, role_id=031d4e1a23e34f71858fc4a6dfbd4ccb)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:41:50.635637 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-adcd5695-005b-4fe6-baf1-2375c35f65dd admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:41:50.647898 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1235/2486] 10.222.0.50 () {60 vars in 1483 bytes} [Tue Sep 13 16:41:50 2022] PUT /identity/v3/projects/c2310d76bc7c40d8ac8e1276ce60126f/users/d29755d7854441c8a5c85aa16c7a05f5/roles/15db4aaca3264ae29bde38903ba43471 => generated 0 bytes in 56 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 16:41:50.648635 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f022468f-3c12-483d-8c94-0ffec762926c admin admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:41:50.656609 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1252/2487] 10.222.0.50 () {60 vars in 1483 bytes} [Tue Sep 13 16:41:50 2022] PUT /identity/v3/projects/da3174247d754b528afbc974e8f9bc31/users/181b443239e8473b841ca01c13d1e446/roles/031d4e1a23e34f71858fc4a6dfbd4ccb => generated 0 bytes in 46 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 16:41:50.658225 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0556a631-a859-4593-8f6a-7af2d8643e23 None None] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:41:50.658515 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0556a631-a859-4593-8f6a-7af2d8643e23 None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:41:50.658794 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0556a631-a859-4593-8f6a-7af2d8643e23 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:41:50.690044 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.auth.core [None req-0556a631-a859-4593-8f6a-7af2d8643e23 None None] MFA Rules not processed for user `d29755d7854441c8a5c85aa16c7a05f5`. Rule list: `[]` (Enabled: `True`). {{(pid=81437) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:41:50.691173 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-0556a631-a859-4593-8f6a-7af2d8643e23 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:41:50.728434 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f022468f-3c12-483d-8c94-0ffec762926c admin admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:41:50.789584 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1253/2488] 10.222.0.50 () {58 vars in 958 bytes} [Tue Sep 13 16:41:50 2022] POST /identity/v3/auth/tokens => generated 3264 bytes in 132 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:41:50.794256 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0556a631-a859-4593-8f6a-7af2d8643e23 None None] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:41:50.804060 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-010efe76-110a-4a7c-bcf4-02828f3aefec admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:41:50.804821 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-010efe76-110a-4a7c-bcf4-02828f3aefec admin admin] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:41:50.805190 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-010efe76-110a-4a7c-bcf4-02828f3aefec admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:41:50.805721 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-010efe76-110a-4a7c-bcf4-02828f3aefec admin admin] PATH_INFO: `/v3/projects` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:41:50.807049 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-010efe76-110a-4a7c-bcf4-02828f3aefec admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:41:50.809702 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-010efe76-110a-4a7c-bcf4-02828f3aefec admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:41:50.849299 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1236/2489] 10.222.0.50 () {60 vars in 1153 bytes} [Tue Sep 13 16:41:50 2022] POST /identity/v3/projects => generated 367 bytes in 201 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 16:41:50.854687 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-010efe76-110a-4a7c-bcf4-02828f3aefec admin admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:41:50.863548 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:INSERT=2,keystone:SELECT=98 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:41:50.872243 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:41:50.885297 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0556a631-a859-4593-8f6a-7af2d8643e23 None None] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:41:50.951549 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-010efe76-110a-4a7c-bcf4-02828f3aefec admin admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:41:50.959162 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33f72b07-57b9-43a9-a651-6e7fd0fc441a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:41:50.960011 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33f72b07-57b9-43a9-a651-6e7fd0fc441a admin admin] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:41:50.960411 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33f72b07-57b9-43a9-a651-6e7fd0fc441a admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:41:50.960796 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33f72b07-57b9-43a9-a651-6e7fd0fc441a admin admin] PATH_INFO: `/v3/users` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:41:50.961524 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33f72b07-57b9-43a9-a651-6e7fd0fc441a admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:41:50.964317 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33f72b07-57b9-43a9-a651-6e7fd0fc441a admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:41:50.972421 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bf985e9-603f-43d3-851a-bc86f21a01b4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:41:50.973209 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bf985e9-603f-43d3-851a-bc86f21a01b4 admin admin] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:41:50.973552 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bf985e9-603f-43d3-851a-bc86f21a01b4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:41:50.973894 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bf985e9-603f-43d3-851a-bc86f21a01b4 admin admin] PATH_INFO: `/v3/projects` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:41:50.974773 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1bf985e9-603f-43d3-851a-bc86f21a01b4 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:41:50.977708 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1bf985e9-603f-43d3-851a-bc86f21a01b4 admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:41:51.002163 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1237/2490] 10.222.0.50 () {60 vars in 1144 bytes} [Tue Sep 13 16:41:50 2022] POST /identity/v3/users => generated 284 bytes in 144 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 16:41:51.003865 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33f72b07-57b9-43a9-a651-6e7fd0fc441a admin admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:41:51.015590 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33f72b07-57b9-43a9-a651-6e7fd0fc441a admin admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:41:51.023967 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1254/2491] 10.222.0.50 () {60 vars in 1153 bytes} [Tue Sep 13 16:41:50 2022] POST /identity/v3/projects => generated 387 bytes in 227 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 13 16:41:51.024160 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60113b6d-5117-4fb2-9311-1f97fa661600 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:41:51.024160 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60113b6d-5117-4fb2-9311-1f97fa661600 admin admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:41:51.024160 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60113b6d-5117-4fb2-9311-1f97fa661600 admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:41:51.024160 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60113b6d-5117-4fb2-9311-1f97fa661600 admin admin] PATH_INFO: `/v3/roles` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:41:51.024470 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60113b6d-5117-4fb2-9311-1f97fa661600 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:41:51.025449 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60113b6d-5117-4fb2-9311-1f97fa661600 admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:41:51.025911 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bf985e9-603f-43d3-851a-bc86f21a01b4 admin admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:41:51.044757 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1238/2492] 10.222.0.50 () {58 vars in 1122 bytes} [Tue Sep 13 16:41:51 2022] GET /identity/v3/roles => generated 1420 bytes in 42 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 13 16:41:51.049611 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60113b6d-5117-4fb2-9311-1f97fa661600 admin admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:41:51.062450 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60113b6d-5117-4fb2-9311-1f97fa661600 admin admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:41:51.154146 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bf985e9-603f-43d3-851a-bc86f21a01b4 admin admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:41:51.162507 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-949a8148-8ab3-4bf5-a531-339318056b5e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:41:51.163166 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-949a8148-8ab3-4bf5-a531-339318056b5e admin admin] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:41:51.163459 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-949a8148-8ab3-4bf5-a531-339318056b5e admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:41:51.163716 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-949a8148-8ab3-4bf5-a531-339318056b5e admin admin] PATH_INFO: `/v3/users` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:41:51.164355 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-949a8148-8ab3-4bf5-a531-339318056b5e admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:41:51.166325 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-949a8148-8ab3-4bf5-a531-339318056b5e admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:41:51.181422 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3642ded-813b-48f6-aa7c-9f31dd473df7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:41:51.182675 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3642ded-813b-48f6-aa7c-9f31dd473df7 admin admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:41:51.183390 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3642ded-813b-48f6-aa7c-9f31dd473df7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:41:51.183827 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3642ded-813b-48f6-aa7c-9f31dd473df7 admin admin] PATH_INFO: `/v3/roles` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:41:51.184539 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3642ded-813b-48f6-aa7c-9f31dd473df7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:41:51.186986 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3642ded-813b-48f6-aa7c-9f31dd473df7 admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:41:51.201543 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1255/2493] 10.222.0.50 () {60 vars in 1144 bytes} [Tue Sep 13 16:41:51 2022] POST /identity/v3/users => generated 294 bytes in 173 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 13 16:41:51.202547 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1239/2494] 10.222.0.50 () {58 vars in 1122 bytes} [Tue Sep 13 16:41:51 2022] GET /identity/v3/roles => generated 1420 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 13 16:41:51.208089 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3642ded-813b-48f6-aa7c-9f31dd473df7 admin admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:41:51.208607 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-949a8148-8ab3-4bf5-a531-339318056b5e admin admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:41:51.222641 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-949a8148-8ab3-4bf5-a531-339318056b5e admin admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:41:51.223592 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3642ded-813b-48f6-aa7c-9f31dd473df7 admin admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:41:51.228657 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29148b80-a30a-45bc-acd1-2781ae32dbf4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:41:51.229198 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83240ab7-e958-4e50-9d0c-19e01e6fe2de admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:41:51.229471 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29148b80-a30a-45bc-acd1-2781ae32dbf4 admin admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:41:51.229760 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29148b80-a30a-45bc-acd1-2781ae32dbf4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:41:51.230040 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29148b80-a30a-45bc-acd1-2781ae32dbf4 admin admin] PATH_INFO: `/v3/roles` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:41:51.230693 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29148b80-a30a-45bc-acd1-2781ae32dbf4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:41:51.230928 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83240ab7-e958-4e50-9d0c-19e01e6fe2de admin admin] REQUEST_METHOD: `PUT` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:41:51.231413 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83240ab7-e958-4e50-9d0c-19e01e6fe2de admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:41:51.231887 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83240ab7-e958-4e50-9d0c-19e01e6fe2de admin admin] PATH_INFO: `/v3/projects/76f13fd33b234c52bc54b2d07daf2ea9/users/8a0d1e23c67a4ba48fff34d6ecdadf38/roles/15db4aaca3264ae29bde38903ba43471` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:41:51.232409 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29148b80-a30a-45bc-acd1-2781ae32dbf4 admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:41:51.247463 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1256/2495] 10.222.0.50 () {58 vars in 1122 bytes} [Tue Sep 13 16:41:51 2022] GET /identity/v3/roles => generated 1420 bytes in 40 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 16:41:51.252888 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29148b80-a30a-45bc-acd1-2781ae32dbf4 admin admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:41:51.259261 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83240ab7-e958-4e50-9d0c-19e01e6fe2de admin admin] RBAC: Authorizing `identity:create_grant(project_id=76f13fd33b234c52bc54b2d07daf2ea9, user_id=8a0d1e23c67a4ba48fff34d6ecdadf38, role_id=15db4aaca3264ae29bde38903ba43471)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:41:51.262597 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83240ab7-e958-4e50-9d0c-19e01e6fe2de admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:41:51.265426 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29148b80-a30a-45bc-acd1-2781ae32dbf4 admin admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:41:51.271824 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00e2eff7-2023-470d-a4e8-f8c72209a5bc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:41:51.272738 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00e2eff7-2023-470d-a4e8-f8c72209a5bc admin admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:41:51.272738 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00e2eff7-2023-470d-a4e8-f8c72209a5bc admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:41:51.273005 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00e2eff7-2023-470d-a4e8-f8c72209a5bc admin admin] PATH_INFO: `/v3/roles` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:41:51.273620 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00e2eff7-2023-470d-a4e8-f8c72209a5bc admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:41:51.275436 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00e2eff7-2023-470d-a4e8-f8c72209a5bc admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:41:51.290645 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1257/2496] 10.222.0.50 () {58 vars in 1122 bytes} [Tue Sep 13 16:41:51 2022] GET /identity/v3/roles => generated 1420 bytes in 40 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 16:41:51.295819 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00e2eff7-2023-470d-a4e8-f8c72209a5bc admin admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:41:51.304273 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1240/2497] 10.222.0.50 () {60 vars in 1483 bytes} [Tue Sep 13 16:41:51 2022] PUT /identity/v3/projects/76f13fd33b234c52bc54b2d07daf2ea9/users/8a0d1e23c67a4ba48fff34d6ecdadf38/roles/15db4aaca3264ae29bde38903ba43471 => generated 0 bytes in 97 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 16:41:51.308379 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00e2eff7-2023-470d-a4e8-f8c72209a5bc admin admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:41:51.314658 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a58bdd5-dc06-4581-b48a-6d5afb8b592a None None] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:41:51.315230 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a58bdd5-dc06-4581-b48a-6d5afb8b592a None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:41:51.315406 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a58bdd5-dc06-4581-b48a-6d5afb8b592a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:41:51.361364 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.auth.core [None req-6a58bdd5-dc06-4581-b48a-6d5afb8b592a None None] MFA Rules not processed for user `8a0d1e23c67a4ba48fff34d6ecdadf38`. Rule list: `[]` (Enabled: `True`). {{(pid=81438) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:41:51.365637 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-6a58bdd5-dc06-4581-b48a-6d5afb8b592a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:41:51.387163 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:41:51.404731 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8130b662-cb5c-4131-bc39-0a08a9d75a96 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:41:51.405709 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8130b662-cb5c-4131-bc39-0a08a9d75a96 admin admin] REQUEST_METHOD: `PUT` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:41:51.406170 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8130b662-cb5c-4131-bc39-0a08a9d75a96 admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:41:51.406765 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8130b662-cb5c-4131-bc39-0a08a9d75a96 admin admin] PATH_INFO: `/v3/projects/ccabb33344594dfda157d06b1cb5342e/users/26066ccbf514493997feeb5892f823e6/roles/15db4aaca3264ae29bde38903ba43471` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:41:51.432507 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8130b662-cb5c-4131-bc39-0a08a9d75a96 admin admin] RBAC: Authorizing `identity:create_grant(project_id=ccabb33344594dfda157d06b1cb5342e, user_id=26066ccbf514493997feeb5892f823e6, role_id=15db4aaca3264ae29bde38903ba43471)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:41:51.436034 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8130b662-cb5c-4131-bc39-0a08a9d75a96 admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:41:51.450343 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:41:51.458351 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1258/2498] 10.222.0.50 () {60 vars in 1483 bytes} [Tue Sep 13 16:41:51 2022] PUT /identity/v3/projects/ccabb33344594dfda157d06b1cb5342e/users/26066ccbf514493997feeb5892f823e6/roles/15db4aaca3264ae29bde38903ba43471 => generated 0 bytes in 163 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 16:41:51.465059 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1241/2499] 10.222.0.50 () {58 vars in 958 bytes} [Tue Sep 13 16:41:51 2022] POST /identity/v3/auth/tokens => generated 3244 bytes in 151 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 13 16:41:51.470336 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a58bdd5-dc06-4581-b48a-6d5afb8b592a None None] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:41:51.471720 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c5f7594-48ec-442d-a45c-08c2c8d8c216 None None] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:41:51.472193 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c5f7594-48ec-442d-a45c-08c2c8d8c216 None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:41:51.473469 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c5f7594-48ec-442d-a45c-08c2c8d8c216 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:41:51.493092 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a58bdd5-dc06-4581-b48a-6d5afb8b592a None None] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:41:51.554872 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.auth.core [None req-9c5f7594-48ec-442d-a45c-08c2c8d8c216 None None] MFA Rules not processed for user `26066ccbf514493997feeb5892f823e6`. Rule list: `[]` (Enabled: `True`). {{(pid=81437) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:41:51.559339 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-9c5f7594-48ec-442d-a45c-08c2c8d8c216 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:41:51.596848 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3eaaae6-0eb2-458a-add0-c68122a6d885 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:41:51.597718 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3eaaae6-0eb2-458a-add0-c68122a6d885 admin admin] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:41:51.598099 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3eaaae6-0eb2-458a-add0-c68122a6d885 admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:41:51.598503 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3eaaae6-0eb2-458a-add0-c68122a6d885 admin admin] PATH_INFO: `/v3/projects` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:41:51.602265 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3eaaae6-0eb2-458a-add0-c68122a6d885 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:41:51.607181 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3eaaae6-0eb2-458a-add0-c68122a6d885 admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:41:51.668287 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1242/2500] 10.222.0.50 () {60 vars in 1153 bytes} [Tue Sep 13 16:41:51 2022] POST /identity/v3/projects => generated 369 bytes in 199 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 16:41:51.669436 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3eaaae6-0eb2-458a-add0-c68122a6d885 admin admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:41:51.677498 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1259/2501] 10.222.0.50 () {58 vars in 958 bytes} [Tue Sep 13 16:41:51 2022] POST /identity/v3/auth/tokens => generated 3264 bytes in 208 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:41:51.681844 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c5f7594-48ec-442d-a45c-08c2c8d8c216 None None] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:41:51.701301 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c5f7594-48ec-442d-a45c-08c2c8d8c216 None None] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:41:51.779054 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3eaaae6-0eb2-458a-add0-c68122a6d885 admin admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:41:51.794706 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b956e01a-40a1-4b97-ad23-1a437ebb38f8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:41:51.795505 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b956e01a-40a1-4b97-ad23-1a437ebb38f8 admin admin] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:41:51.795791 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b956e01a-40a1-4b97-ad23-1a437ebb38f8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:41:51.796562 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b956e01a-40a1-4b97-ad23-1a437ebb38f8 admin admin] PATH_INFO: `/v3/users` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:41:51.798504 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b956e01a-40a1-4b97-ad23-1a437ebb38f8 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:41:51.801959 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b956e01a-40a1-4b97-ad23-1a437ebb38f8 admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:41:51.827813 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86e3c966-011c-45dd-99d3-e75a7558173c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:41:51.828832 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86e3c966-011c-45dd-99d3-e75a7558173c admin admin] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:41:51.829835 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86e3c966-011c-45dd-99d3-e75a7558173c admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:41:51.830637 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86e3c966-011c-45dd-99d3-e75a7558173c admin admin] PATH_INFO: `/v3/projects` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:41:51.831938 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86e3c966-011c-45dd-99d3-e75a7558173c admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:41:51.834649 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86e3c966-011c-45dd-99d3-e75a7558173c admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:41:51.875042 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1260/2502] 10.222.0.50 () {60 vars in 1153 bytes} [Tue Sep 13 16:41:51 2022] POST /identity/v3/projects => generated 387 bytes in 193 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 13 16:41:51.875970 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86e3c966-011c-45dd-99d3-e75a7558173c admin admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:41:51.880024 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1243/2503] 10.222.0.50 () {60 vars in 1144 bytes} [Tue Sep 13 16:41:51 2022] POST /identity/v3/users => generated 291 bytes in 209 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 16:41:51.880956 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b956e01a-40a1-4b97-ad23-1a437ebb38f8 admin admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:41:51.916487 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b956e01a-40a1-4b97-ad23-1a437ebb38f8 admin admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:41:51.949415 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:41:51.965736 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86e3c966-011c-45dd-99d3-e75a7558173c admin admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:41:51.971602 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c36de401-e405-4874-943a-b8697a019802 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:41:51.972325 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c36de401-e405-4874-943a-b8697a019802 admin admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:41:51.972676 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c36de401-e405-4874-943a-b8697a019802 admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:41:51.973045 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c36de401-e405-4874-943a-b8697a019802 admin admin] PATH_INFO: `/v3/roles` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:41:51.974030 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c36de401-e405-4874-943a-b8697a019802 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:41:51.976414 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c36de401-e405-4874-943a-b8697a019802 admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:41:51.998968 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1261/2504] 10.222.0.50 () {58 vars in 1122 bytes} [Tue Sep 13 16:41:51 2022] GET /identity/v3/roles => generated 1420 bytes in 116 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 16:41:51.998968 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c36de401-e405-4874-943a-b8697a019802 admin admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:41:52.021453 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c36de401-e405-4874-943a-b8697a019802 admin admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:41:52.032275 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64124ec6-83ea-4a0e-9539-e202c9265999 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:41:52.033552 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64124ec6-83ea-4a0e-9539-e202c9265999 admin admin] REQUEST_METHOD: `PUT` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:41:52.034212 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64124ec6-83ea-4a0e-9539-e202c9265999 admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:41:52.034212 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64124ec6-83ea-4a0e-9539-e202c9265999 admin admin] PATH_INFO: `/v3/projects/212d2d05a3dc4b82abc868f3766ac064/users/6b8d33103e254450b75c92365f7673fd/roles/031d4e1a23e34f71858fc4a6dfbd4ccb` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:41:52.056890 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12200195-445f-472c-8089-eee704c83a11 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:41:52.057677 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12200195-445f-472c-8089-eee704c83a11 admin admin] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:41:52.058165 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12200195-445f-472c-8089-eee704c83a11 admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:41:52.058519 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12200195-445f-472c-8089-eee704c83a11 admin admin] PATH_INFO: `/v3/users` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:41:52.059892 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12200195-445f-472c-8089-eee704c83a11 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:41:52.066078 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12200195-445f-472c-8089-eee704c83a11 admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:41:52.099601 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=93,keystone:INSERT=7 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:41:52.105515 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64124ec6-83ea-4a0e-9539-e202c9265999 admin admin] RBAC: Authorizing `identity:create_grant(project_id=212d2d05a3dc4b82abc868f3766ac064, user_id=6b8d33103e254450b75c92365f7673fd, role_id=031d4e1a23e34f71858fc4a6dfbd4ccb)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:41:52.112371 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64124ec6-83ea-4a0e-9539-e202c9265999 admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:41:52.112667 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1244/2505] 10.222.0.50 () {60 vars in 1144 bytes} [Tue Sep 13 16:41:51 2022] POST /identity/v3/users => generated 300 bytes in 232 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 16:41:52.116793 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12200195-445f-472c-8089-eee704c83a11 admin admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:41:52.132743 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12200195-445f-472c-8089-eee704c83a11 admin admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:41:52.134242 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1262/2506] 10.222.0.50 () {60 vars in 1483 bytes} [Tue Sep 13 16:41:51 2022] PUT /identity/v3/projects/212d2d05a3dc4b82abc868f3766ac064/users/6b8d33103e254450b75c92365f7673fd/roles/031d4e1a23e34f71858fc4a6dfbd4ccb => generated 0 bytes in 138 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 16:41:52.138165 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64124ec6-83ea-4a0e-9539-e202c9265999 admin admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:41:52.230143 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfe9b9d2-f6ea-4247-a925-225ee60fb6aa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:41:52.230985 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfe9b9d2-f6ea-4247-a925-225ee60fb6aa admin admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:41:52.231291 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfe9b9d2-f6ea-4247-a925-225ee60fb6aa admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:41:52.231733 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfe9b9d2-f6ea-4247-a925-225ee60fb6aa admin admin] PATH_INFO: `/v3/roles` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:41:52.232396 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dfe9b9d2-f6ea-4247-a925-225ee60fb6aa admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:41:52.234020 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dfe9b9d2-f6ea-4247-a925-225ee60fb6aa admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:41:52.255303 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1245/2507] 10.222.0.50 () {58 vars in 1122 bytes} [Tue Sep 13 16:41:52 2022] GET /identity/v3/roles => generated 1420 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 16:41:52.260541 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfe9b9d2-f6ea-4247-a925-225ee60fb6aa admin admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:41:52.263320 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64124ec6-83ea-4a0e-9539-e202c9265999 admin admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:41:52.275382 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b51a2b9-6262-4b8b-a018-28d59275f749 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:41:52.276007 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b51a2b9-6262-4b8b-a018-28d59275f749 admin admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:41:52.276288 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b51a2b9-6262-4b8b-a018-28d59275f749 admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:41:52.276555 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b51a2b9-6262-4b8b-a018-28d59275f749 admin admin] PATH_INFO: `/v3/roles` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:41:52.281182 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b51a2b9-6262-4b8b-a018-28d59275f749 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:41:52.285369 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b51a2b9-6262-4b8b-a018-28d59275f749 admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:41:52.286439 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfe9b9d2-f6ea-4247-a925-225ee60fb6aa admin admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:41:52.294985 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f6d3721-bffa-4170-818b-61404fb12a9c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:41:52.295672 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f6d3721-bffa-4170-818b-61404fb12a9c admin admin] REQUEST_METHOD: `PUT` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:41:52.295959 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f6d3721-bffa-4170-818b-61404fb12a9c admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:41:52.296242 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f6d3721-bffa-4170-818b-61404fb12a9c admin admin] PATH_INFO: `/v3/projects/f2dd36092ffe45eb9fe245ec2af9d3a9/users/382addbce3f04db8a648ecaf2a23a4f2/roles/031d4e1a23e34f71858fc4a6dfbd4ccb` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:41:52.306097 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1263/2508] 10.222.0.50 () {58 vars in 1122 bytes} [Tue Sep 13 16:41:52 2022] GET /identity/v3/roles => generated 1420 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 13 16:41:52.307489 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b51a2b9-6262-4b8b-a018-28d59275f749 admin admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:41:52.317339 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f6d3721-bffa-4170-818b-61404fb12a9c admin admin] RBAC: Authorizing `identity:create_grant(project_id=f2dd36092ffe45eb9fe245ec2af9d3a9, user_id=382addbce3f04db8a648ecaf2a23a4f2, role_id=031d4e1a23e34f71858fc4a6dfbd4ccb)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:41:52.322352 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f6d3721-bffa-4170-818b-61404fb12a9c admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:41:52.322916 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b51a2b9-6262-4b8b-a018-28d59275f749 admin admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:41:52.342350 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54e387b4-4cf5-492b-aa79-885767c43aa2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:41:52.343549 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54e387b4-4cf5-492b-aa79-885767c43aa2 admin admin] REQUEST_METHOD: `PUT` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:41:52.343882 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54e387b4-4cf5-492b-aa79-885767c43aa2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:41:52.344258 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54e387b4-4cf5-492b-aa79-885767c43aa2 admin admin] PATH_INFO: `/v3/projects/212d2d05a3dc4b82abc868f3766ac064/users/6b8d33103e254450b75c92365f7673fd/roles/031d4e1a23e34f71858fc4a6dfbd4ccb` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:41:52.348023 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1246/2509] 10.222.0.50 () {60 vars in 1483 bytes} [Tue Sep 13 16:41:52 2022] PUT /identity/v3/projects/f2dd36092ffe45eb9fe245ec2af9d3a9/users/382addbce3f04db8a648ecaf2a23a4f2/roles/031d4e1a23e34f71858fc4a6dfbd4ccb => generated 0 bytes in 85 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 16:41:52.349683 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54e387b4-4cf5-492b-aa79-885767c43aa2 admin admin] RBAC: Authorizing `identity:create_grant(project_id=212d2d05a3dc4b82abc868f3766ac064, user_id=6b8d33103e254450b75c92365f7673fd, role_id=031d4e1a23e34f71858fc4a6dfbd4ccb)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:41:52.353119 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54e387b4-4cf5-492b-aa79-885767c43aa2 admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:41:52.356868 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f6d3721-bffa-4170-818b-61404fb12a9c admin admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:41:52.384361 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1264/2510] 10.222.0.50 () {60 vars in 1483 bytes} [Tue Sep 13 16:41:52 2022] PUT /identity/v3/projects/212d2d05a3dc4b82abc868f3766ac064/users/6b8d33103e254450b75c92365f7673fd/roles/031d4e1a23e34f71858fc4a6dfbd4ccb => generated 0 bytes in 76 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 16:41:52.395043 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5474244-d591-415a-81f2-47a0194478e0 None None] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:41:52.395332 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5474244-d591-415a-81f2-47a0194478e0 None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:41:52.396999 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5474244-d591-415a-81f2-47a0194478e0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:41:52.448985 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.auth.core [None req-e5474244-d591-415a-81f2-47a0194478e0 None None] MFA Rules not processed for user `6b8d33103e254450b75c92365f7673fd`. Rule list: `[]` (Enabled: `True`). {{(pid=81437) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:41:52.451508 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-e5474244-d591-415a-81f2-47a0194478e0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:41:52.505403 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f6d3721-bffa-4170-818b-61404fb12a9c admin admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:41:52.611452 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65fc7a7a-a8d9-4bf9-9847-adab63dae47b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:41:52.612079 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65fc7a7a-a8d9-4bf9-9847-adab63dae47b admin admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:41:52.612361 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65fc7a7a-a8d9-4bf9-9847-adab63dae47b admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:41:52.612727 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65fc7a7a-a8d9-4bf9-9847-adab63dae47b admin admin] PATH_INFO: `/v3/roles` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:41:52.613382 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65fc7a7a-a8d9-4bf9-9847-adab63dae47b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:41:52.615694 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65fc7a7a-a8d9-4bf9-9847-adab63dae47b admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:41:52.623430 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1265/2511] 10.222.0.50 () {58 vars in 958 bytes} [Tue Sep 13 16:41:52 2022] POST /identity/v3/auth/tokens => generated 3313 bytes in 230 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:41:52.636185 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5474244-d591-415a-81f2-47a0194478e0 None None] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:41:52.637809 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-e5474244-d591-415a-81f2-47a0194478e0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:41:52.638576 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1247/2512] 10.222.0.50 () {58 vars in 1122 bytes} [Tue Sep 13 16:41:52 2022] GET /identity/v3/roles => generated 1420 bytes in 283 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 16:41:52.642806 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65fc7a7a-a8d9-4bf9-9847-adab63dae47b admin admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:41:52.654002 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65fc7a7a-a8d9-4bf9-9847-adab63dae47b admin admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:41:52.660442 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25a830d9-17b9-46e8-afc7-2ff29f75d9d5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:41:52.661638 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25a830d9-17b9-46e8-afc7-2ff29f75d9d5 admin admin] REQUEST_METHOD: `PUT` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:41:52.661991 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25a830d9-17b9-46e8-afc7-2ff29f75d9d5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:41:52.662480 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25a830d9-17b9-46e8-afc7-2ff29f75d9d5 admin admin] PATH_INFO: `/v3/projects/f2dd36092ffe45eb9fe245ec2af9d3a9/users/382addbce3f04db8a648ecaf2a23a4f2/roles/031d4e1a23e34f71858fc4a6dfbd4ccb` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:41:52.664730 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25a830d9-17b9-46e8-afc7-2ff29f75d9d5 admin admin] RBAC: Authorizing `identity:create_grant(project_id=f2dd36092ffe45eb9fe245ec2af9d3a9, user_id=382addbce3f04db8a648ecaf2a23a4f2, role_id=031d4e1a23e34f71858fc4a6dfbd4ccb)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:41:52.667740 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25a830d9-17b9-46e8-afc7-2ff29f75d9d5 admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:41:52.676310 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:41:52.687866 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1248/2513] 10.222.0.50 () {60 vars in 1483 bytes} [Tue Sep 13 16:41:52 2022] PUT /identity/v3/projects/f2dd36092ffe45eb9fe245ec2af9d3a9/users/382addbce3f04db8a648ecaf2a23a4f2/roles/031d4e1a23e34f71858fc4a6dfbd4ccb => generated 0 bytes in 46 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 16:41:52.693074 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:41:52.698982 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8519238c-2aff-4ba5-94d2-2cefb0a1b9ed None None] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:41:52.699273 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8519238c-2aff-4ba5-94d2-2cefb0a1b9ed None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:41:52.699601 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8519238c-2aff-4ba5-94d2-2cefb0a1b9ed None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:41:52.727893 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5474244-d591-415a-81f2-47a0194478e0 None None] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:41:52.761368 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.auth.core [None req-8519238c-2aff-4ba5-94d2-2cefb0a1b9ed None None] MFA Rules not processed for user `382addbce3f04db8a648ecaf2a23a4f2`. Rule list: `[]` (Enabled: `True`). {{(pid=81438) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:41:52.765610 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-8519238c-2aff-4ba5-94d2-2cefb0a1b9ed None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:41:52.832337 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04b0df49-1845-4c56-a49f-4b320b15cc53 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7fd6b5afd67b48f9b0f0597ccd4cf4df', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:41:52.833196 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04b0df49-1845-4c56-a49f-4b320b15cc53 service cinder] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:41:52.833618 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04b0df49-1845-4c56-a49f-4b320b15cc53 service cinder] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:41:52.834258 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04b0df49-1845-4c56-a49f-4b320b15cc53 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:41:52.909673 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1249/2514] 10.222.0.50 () {58 vars in 958 bytes} [Tue Sep 13 16:41:52 2022] POST /identity/v3/auth/tokens => generated 3331 bytes in 209 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 13 16:41:52.920148 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8519238c-2aff-4ba5-94d2-2cefb0a1b9ed None None] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:41:52.921753 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-8519238c-2aff-4ba5-94d2-2cefb0a1b9ed None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:41:52.927787 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04b0df49-1845-4c56-a49f-4b320b15cc53 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:41:52.929566 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04b0df49-1845-4c56-a49f-4b320b15cc53 service cinder] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:41:52.944965 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8519238c-2aff-4ba5-94d2-2cefb0a1b9ed None None] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:41:52.946704 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1266/2515] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 16:41:52 2022] GET /identity/v3/auth/tokens => generated 3313 bytes in 311 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 13 16:41:52.949769 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d0a53e6-5d78-4806-b3ce-f0e03ec2f24c service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7fd6b5afd67b48f9b0f0597ccd4cf4df', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:41:52.950616 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d0a53e6-5d78-4806-b3ce-f0e03ec2f24c service cinder] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:41:52.951098 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d0a53e6-5d78-4806-b3ce-f0e03ec2f24c service cinder] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:41:52.951563 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d0a53e6-5d78-4806-b3ce-f0e03ec2f24c service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:41:52.965809 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d0a53e6-5d78-4806-b3ce-f0e03ec2f24c service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:41:52.967309 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d0a53e6-5d78-4806-b3ce-f0e03ec2f24c service cinder] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:41:52.983193 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1250/2516] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 16:41:52 2022] GET /identity/v3/auth/tokens => generated 3331 bytes in 64 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 16:41:53.047004 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04b0df49-1845-4c56-a49f-4b320b15cc53 service cinder] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:41:53.065598 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04b0df49-1845-4c56-a49f-4b320b15cc53 service cinder] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:41:53.069604 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6e97079-06aa-48fc-ad4e-5d4257975114 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7fd6b5afd67b48f9b0f0597ccd4cf4df', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:41:53.070227 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6e97079-06aa-48fc-ad4e-5d4257975114 service cinder] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:41:53.070529 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6e97079-06aa-48fc-ad4e-5d4257975114 service cinder] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:41:53.070797 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6e97079-06aa-48fc-ad4e-5d4257975114 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:41:53.179980 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6e97079-06aa-48fc-ad4e-5d4257975114 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:41:53.181530 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6e97079-06aa-48fc-ad4e-5d4257975114 service cinder] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:41:53.198053 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1267/2517] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 16:41:53 2022] GET /identity/v3/auth/tokens => generated 3244 bytes in 152 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 16:41:54.228529 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d0a53e6-5d78-4806-b3ce-f0e03ec2f24c service cinder] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:41:54.242673 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d0a53e6-5d78-4806-b3ce-f0e03ec2f24c service cinder] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:41:54.247541 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e2305b4-2a0f-4ef6-a5b2-bcf958a65b59 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7fd6b5afd67b48f9b0f0597ccd4cf4df', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:41:54.248276 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e2305b4-2a0f-4ef6-a5b2-bcf958a65b59 service cinder] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:41:54.248640 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e2305b4-2a0f-4ef6-a5b2-bcf958a65b59 service cinder] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:41:54.248990 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e2305b4-2a0f-4ef6-a5b2-bcf958a65b59 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:41:54.334579 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e2305b4-2a0f-4ef6-a5b2-bcf958a65b59 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:41:54.336314 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e2305b4-2a0f-4ef6-a5b2-bcf958a65b59 service cinder] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:41:54.350788 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1251/2518] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 16:41:54 2022] GET /identity/v3/auth/tokens => generated 3264 bytes in 123 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 16:41:56.195727 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df648e9c-bd4e-483d-8a88-c22cb354f60e None None] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:41:56.196206 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df648e9c-bd4e-483d-8a88-c22cb354f60e None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:41:56.196327 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df648e9c-bd4e-483d-8a88-c22cb354f60e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:41:56.226379 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.auth.core [None req-df648e9c-bd4e-483d-8a88-c22cb354f60e None None] MFA Rules not processed for user `181b443239e8473b841ca01c13d1e446`. Rule list: `[]` (Enabled: `True`). {{(pid=81437) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:41:56.228507 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-df648e9c-bd4e-483d-8a88-c22cb354f60e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:41:56.257672 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:41:56.303830 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1268/2519] 10.222.0.50 () {58 vars in 958 bytes} [Tue Sep 13 16:41:56 2022] POST /identity/v3/auth/tokens => generated 3323 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:41:56.320204 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e2305b4-2a0f-4ef6-a5b2-bcf958a65b59 service cinder] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:41:56.321796 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-8e2305b4-2a0f-4ef6-a5b2-bcf958a65b59 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:41:56.400404 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e2305b4-2a0f-4ef6-a5b2-bcf958a65b59 service cinder] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:41:56.406114 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ceba2feb-0eb9-49ab-8385-d2646bb38b88 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4e039b46c5e3445bbb39904b00c5430e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'admin', 'ResellerAdmin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:41:56.406814 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ceba2feb-0eb9-49ab-8385-d2646bb38b88 service nova] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:41:56.407128 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ceba2feb-0eb9-49ab-8385-d2646bb38b88 service nova] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:41:56.407968 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ceba2feb-0eb9-49ab-8385-d2646bb38b88 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:41:56.409737 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-ceba2feb-0eb9-49ab-8385-d2646bb38b88 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:41:56.415891 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:41:56.472553 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ceba2feb-0eb9-49ab-8385-d2646bb38b88 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:41:56.474165 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ceba2feb-0eb9-49ab-8385-d2646bb38b88 service nova] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:41:56.483485 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1252/2520] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 16:41:56 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 164 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 16:41:57.551784 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df648e9c-bd4e-483d-8a88-c22cb354f60e None None] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:41:57.668669 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df648e9c-bd4e-483d-8a88-c22cb354f60e None None] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:41:57.674641 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-845347fb-7bbf-4c59-9525-8ad89156be36 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:41:57.675819 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-845347fb-7bbf-4c59-9525-8ad89156be36 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:41:57.675819 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-845347fb-7bbf-4c59-9525-8ad89156be36 admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:41:57.676217 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-845347fb-7bbf-4c59-9525-8ad89156be36 admin admin] PATH_INFO: `/v3/users/8a0d1e23c67a4ba48fff34d6ecdadf38` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:41:57.677620 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-845347fb-7bbf-4c59-9525-8ad89156be36 admin admin] RBAC: Authorizing `identity:delete_user(user_id=8a0d1e23c67a4ba48fff34d6ecdadf38)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:41:57.680198 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-845347fb-7bbf-4c59-9525-8ad89156be36 admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:41:57.769447 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-845347fb-7bbf-4c59-9525-8ad89156be36 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '8a0d1e23c67a4ba48fff34d6ecdadf38'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:41:57.779532 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-845347fb-7bbf-4c59-9525-8ad89156be36 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '8a0d1e23c67a4ba48fff34d6ecdadf38'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:41:57.796731 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-845347fb-7bbf-4c59-9525-8ad89156be36 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '8a0d1e23c67a4ba48fff34d6ecdadf38'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:41:57.797606 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-845347fb-7bbf-4c59-9525-8ad89156be36 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '8a0d1e23c67a4ba48fff34d6ecdadf38'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:41:57.808121 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-845347fb-7bbf-4c59-9525-8ad89156be36 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '8a0d1e23c67a4ba48fff34d6ecdadf38'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:41:57.822444 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1269/2521] 10.222.0.50 () {58 vars in 1224 bytes} [Tue Sep 13 16:41:57 2022] DELETE /identity/v3/users/8a0d1e23c67a4ba48fff34d6ecdadf38 => generated 0 bytes in 268 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 16:41:57.824208 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4650de7-e51c-499d-a864-ac12d4d11efd None None] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:41:57.824669 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4650de7-e51c-499d-a864-ac12d4d11efd None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:41:57.824745 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4650de7-e51c-499d-a864-ac12d4d11efd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:41:57.873932 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.auth.core [None req-a4650de7-e51c-499d-a864-ac12d4d11efd None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81438) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:41:57.878686 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-a4650de7-e51c-499d-a864-ac12d4d11efd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:41:58.026051 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1253/2522] 10.222.0.50 () {58 vars in 958 bytes} [Tue Sep 13 16:41:57 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 203 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 13 16:41:58.048736 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-845347fb-7bbf-4c59-9525-8ad89156be36 admin admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:41:58.050391 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-845347fb-7bbf-4c59-9525-8ad89156be36 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:41:58.177040 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:41:58.219925 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-845347fb-7bbf-4c59-9525-8ad89156be36 admin admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:41:58.233324 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d09365e-4809-41c1-8ec7-f52fc0514707 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f436f07939d4855a21b195bc7566244', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:41:58.233900 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d09365e-4809-41c1-8ec7-f52fc0514707 service neutron] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:41:58.234812 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d09365e-4809-41c1-8ec7-f52fc0514707 service neutron] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:41:58.235146 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d09365e-4809-41c1-8ec7-f52fc0514707 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:41:58.258059 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d09365e-4809-41c1-8ec7-f52fc0514707 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:41:58.259827 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d09365e-4809-41c1-8ec7-f52fc0514707 service neutron] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:41:58.277215 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1270/2523] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 16:41:58 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 229 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 16:41:58.613958 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4650de7-e51c-499d-a864-ac12d4d11efd None None] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:41:58.615624 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-a4650de7-e51c-499d-a864-ac12d4d11efd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:41:58.635648 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4650de7-e51c-499d-a864-ac12d4d11efd None None] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:41:58.641376 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c4ee48d-1ec5-4353-a486-edd297dd4a11 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:41:58.642927 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c4ee48d-1ec5-4353-a486-edd297dd4a11 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:41:58.643264 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c4ee48d-1ec5-4353-a486-edd297dd4a11 admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:41:58.643712 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c4ee48d-1ec5-4353-a486-edd297dd4a11 admin admin] PATH_INFO: `/v3/projects/76f13fd33b234c52bc54b2d07daf2ea9` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:41:58.645224 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c4ee48d-1ec5-4353-a486-edd297dd4a11 admin admin] RBAC: Authorizing `identity:delete_project(project_id=76f13fd33b234c52bc54b2d07daf2ea9)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:41:58.647727 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c4ee48d-1ec5-4353-a486-edd297dd4a11 admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:41:58.716652 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-3c4ee48d-1ec5-4353-a486-edd297dd4a11 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '76f13fd33b234c52bc54b2d07daf2ea9', 'request_id': 'req-3c4ee48d-1ec5-4353-a486-edd297dd4a11'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:41:58.730728 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-3c4ee48d-1ec5-4353-a486-edd297dd4a11 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '76f13fd33b234c52bc54b2d07daf2ea9', 'request_id': 'req-3c4ee48d-1ec5-4353-a486-edd297dd4a11'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:41:58.737646 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-3c4ee48d-1ec5-4353-a486-edd297dd4a11 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '76f13fd33b234c52bc54b2d07daf2ea9', 'request_id': 'req-3c4ee48d-1ec5-4353-a486-edd297dd4a11'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:41:58.752453 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-3c4ee48d-1ec5-4353-a486-edd297dd4a11 admin admin] The token cache is being invalidate because project 76f13fd33b234c52bc54b2d07daf2ea9 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81438) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 13 16:41:58.752794 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-3c4ee48d-1ec5-4353-a486-edd297dd4a11 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:41:58.753891 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-3c4ee48d-1ec5-4353-a486-edd297dd4a11 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:41:58.755911 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1254/2524] 10.222.0.50 () {58 vars in 1233 bytes} [Tue Sep 13 16:41:58 2022] DELETE /identity/v3/projects/76f13fd33b234c52bc54b2d07daf2ea9 => generated 0 bytes in 142 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 16:41:58.760652 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d09365e-4809-41c1-8ec7-f52fc0514707 service neutron] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:41:58.775093 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d09365e-4809-41c1-8ec7-f52fc0514707 service neutron] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:41:58.881930 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ccdb986-ad1a-44c6-9c6d-2df0eafdc980 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:41:58.882585 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ccdb986-ad1a-44c6-9c6d-2df0eafdc980 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:41:58.882878 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ccdb986-ad1a-44c6-9c6d-2df0eafdc980 admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:41:58.883507 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ccdb986-ad1a-44c6-9c6d-2df0eafdc980 admin admin] PATH_INFO: `/v3/users/6b8d33103e254450b75c92365f7673fd` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:41:58.884917 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ccdb986-ad1a-44c6-9c6d-2df0eafdc980 admin admin] RBAC: Authorizing `identity:delete_user(user_id=6b8d33103e254450b75c92365f7673fd)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:41:58.887921 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ccdb986-ad1a-44c6-9c6d-2df0eafdc980 admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:41:58.956502 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-9ccdb986-ad1a-44c6-9c6d-2df0eafdc980 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '6b8d33103e254450b75c92365f7673fd'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:41:58.966000 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-9ccdb986-ad1a-44c6-9c6d-2df0eafdc980 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '6b8d33103e254450b75c92365f7673fd'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:41:58.986826 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-9ccdb986-ad1a-44c6-9c6d-2df0eafdc980 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '6b8d33103e254450b75c92365f7673fd'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:41:58.987805 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-9ccdb986-ad1a-44c6-9c6d-2df0eafdc980 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '6b8d33103e254450b75c92365f7673fd'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:41:58.995790 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-9ccdb986-ad1a-44c6-9c6d-2df0eafdc980 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '6b8d33103e254450b75c92365f7673fd'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:41:59.003452 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1271/2525] 10.222.0.50 () {58 vars in 1224 bytes} [Tue Sep 13 16:41:58 2022] DELETE /identity/v3/users/6b8d33103e254450b75c92365f7673fd => generated 0 bytes in 244 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 13 16:41:59.306790 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c4ee48d-1ec5-4353-a486-edd297dd4a11 admin admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:41:59.309253 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-3c4ee48d-1ec5-4353-a486-edd297dd4a11 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:41:59.324859 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c4ee48d-1ec5-4353-a486-edd297dd4a11 admin admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:41:59.374952 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:41:59.412740 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eed37975-c8a9-4463-80e0-e89a78e1ec26 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:41:59.413334 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eed37975-c8a9-4463-80e0-e89a78e1ec26 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:41:59.413622 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eed37975-c8a9-4463-80e0-e89a78e1ec26 admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:41:59.413901 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eed37975-c8a9-4463-80e0-e89a78e1ec26 admin admin] PATH_INFO: `/v3/projects/212d2d05a3dc4b82abc868f3766ac064` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:41:59.415076 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eed37975-c8a9-4463-80e0-e89a78e1ec26 admin admin] RBAC: Authorizing `identity:delete_project(project_id=212d2d05a3dc4b82abc868f3766ac064)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:41:59.417030 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eed37975-c8a9-4463-80e0-e89a78e1ec26 admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:41:59.421520 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ccdb986-ad1a-44c6-9c6d-2df0eafdc980 admin admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:41:59.423174 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-9ccdb986-ad1a-44c6-9c6d-2df0eafdc980 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:41:59.437690 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ccdb986-ad1a-44c6-9c6d-2df0eafdc980 admin admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:41:59.443788 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20224a5c-d9d6-471e-a276-1ec4fdb29b37 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:41:59.444416 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20224a5c-d9d6-471e-a276-1ec4fdb29b37 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:41:59.444697 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20224a5c-d9d6-471e-a276-1ec4fdb29b37 admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:41:59.444963 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20224a5c-d9d6-471e-a276-1ec4fdb29b37 admin admin] PATH_INFO: `/v3/users/78c03dc3584e4918bccf6f08ab00f331` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:41:59.446364 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20224a5c-d9d6-471e-a276-1ec4fdb29b37 admin admin] RBAC: Authorizing `identity:delete_user(user_id=78c03dc3584e4918bccf6f08ab00f331)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:41:59.448785 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20224a5c-d9d6-471e-a276-1ec4fdb29b37 admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:41:59.473863 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-eed37975-c8a9-4463-80e0-e89a78e1ec26 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '212d2d05a3dc4b82abc868f3766ac064', 'request_id': 'req-eed37975-c8a9-4463-80e0-e89a78e1ec26'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:41:59.481876 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-eed37975-c8a9-4463-80e0-e89a78e1ec26 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '212d2d05a3dc4b82abc868f3766ac064', 'request_id': 'req-eed37975-c8a9-4463-80e0-e89a78e1ec26'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:41:59.490641 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-eed37975-c8a9-4463-80e0-e89a78e1ec26 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '212d2d05a3dc4b82abc868f3766ac064', 'request_id': 'req-eed37975-c8a9-4463-80e0-e89a78e1ec26'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:41:59.504748 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-eed37975-c8a9-4463-80e0-e89a78e1ec26 admin admin] The token cache is being invalidate because project 212d2d05a3dc4b82abc868f3766ac064 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81438) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 13 16:41:59.505087 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-20224a5c-d9d6-471e-a276-1ec4fdb29b37 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '78c03dc3584e4918bccf6f08ab00f331'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:41:59.505226 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-eed37975-c8a9-4463-80e0-e89a78e1ec26 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:41:59.510337 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-eed37975-c8a9-4463-80e0-e89a78e1ec26 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:41:59.516658 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1255/2526] 10.222.0.50 () {58 vars in 1233 bytes} [Tue Sep 13 16:41:59 2022] DELETE /identity/v3/projects/212d2d05a3dc4b82abc868f3766ac064 => generated 0 bytes in 207 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 16:41:59.518698 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-20224a5c-d9d6-471e-a276-1ec4fdb29b37 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '78c03dc3584e4918bccf6f08ab00f331'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:41:59.519747 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4af41858-c3b5-4a91-a251-de7180a27dae None None] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:41:59.520262 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4af41858-c3b5-4a91-a251-de7180a27dae None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:41:59.520638 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4af41858-c3b5-4a91-a251-de7180a27dae None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:41:59.521039 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=80,keystone:DELETE=18,keystone:INSERT=2 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:41:59.534617 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-20224a5c-d9d6-471e-a276-1ec4fdb29b37 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '78c03dc3584e4918bccf6f08ab00f331'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:41:59.535360 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-20224a5c-d9d6-471e-a276-1ec4fdb29b37 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '78c03dc3584e4918bccf6f08ab00f331'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:41:59.544733 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-20224a5c-d9d6-471e-a276-1ec4fdb29b37 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '78c03dc3584e4918bccf6f08ab00f331'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:41:59.544893 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.auth.core [None req-4af41858-c3b5-4a91-a251-de7180a27dae None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81438) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:41:59.545944 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-4af41858-c3b5-4a91-a251-de7180a27dae None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:41:59.554950 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1272/2527] 10.222.0.50 () {58 vars in 1224 bytes} [Tue Sep 13 16:41:59 2022] DELETE /identity/v3/users/78c03dc3584e4918bccf6f08ab00f331 => generated 0 bytes in 131 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 13 16:41:59.556394 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dc1e24a-5814-447a-9937-ddf62d19b2b2 None None] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:41:59.556684 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dc1e24a-5814-447a-9937-ddf62d19b2b2 None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:41:59.556963 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dc1e24a-5814-447a-9937-ddf62d19b2b2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:41:59.577548 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.auth.core [None req-8dc1e24a-5814-447a-9937-ddf62d19b2b2 None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81437) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:41:59.578787 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-8dc1e24a-5814-447a-9937-ddf62d19b2b2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:41:59.618361 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1256/2528] 10.222.0.50 () {58 vars in 958 bytes} [Tue Sep 13 16:41:59 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:41:59.627596 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19adc927-6f1f-4062-8502-71c71c5fd75c None None] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:41:59.627901 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19adc927-6f1f-4062-8502-71c71c5fd75c None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:41:59.628218 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19adc927-6f1f-4062-8502-71c71c5fd75c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:41:59.649800 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.auth.core [None req-19adc927-6f1f-4062-8502-71c71c5fd75c None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81438) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:41:59.651042 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-19adc927-6f1f-4062-8502-71c71c5fd75c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:41:59.655497 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1273/2529] 10.222.0.50 () {58 vars in 958 bytes} [Tue Sep 13 16:41:59 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 13 16:41:59.659482 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1257/2530] 10.222.0.50 () {58 vars in 958 bytes} [Tue Sep 13 16:41:59 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:41:59.663895 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8dc1e24a-5814-447a-9937-ddf62d19b2b2 None None] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:41:59.669439 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19adc927-6f1f-4062-8502-71c71c5fd75c None None] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:41:59.670994 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-19adc927-6f1f-4062-8502-71c71c5fd75c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:41:59.679166 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8dc1e24a-5814-447a-9937-ddf62d19b2b2 None None] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:41:59.683887 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86a4a350-f492-424c-a5f1-7a5d5081765d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:41:59.684634 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86a4a350-f492-424c-a5f1-7a5d5081765d admin admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:41:59.684914 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86a4a350-f492-424c-a5f1-7a5d5081765d admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:41:59.685183 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86a4a350-f492-424c-a5f1-7a5d5081765d admin admin] PATH_INFO: `/v3/domains` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:41:59.685827 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86a4a350-f492-424c-a5f1-7a5d5081765d admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81437) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 16:41:59.686148 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86a4a350-f492-424c-a5f1-7a5d5081765d admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:41:59.687805 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86a4a350-f492-424c-a5f1-7a5d5081765d admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:41:59.700502 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1274/2531] 10.222.0.50 () {58 vars in 1153 bytes} [Tue Sep 13 16:41:59 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 16:41:59.705721 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86a4a350-f492-424c-a5f1-7a5d5081765d admin admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:41:59.715814 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86a4a350-f492-424c-a5f1-7a5d5081765d admin admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:41:59.721327 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cab94e85-d79a-46b0-8eb0-8da447c1f5a5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:41:59.721922 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cab94e85-d79a-46b0-8eb0-8da447c1f5a5 admin admin] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:41:59.722198 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cab94e85-d79a-46b0-8eb0-8da447c1f5a5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:41:59.722468 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cab94e85-d79a-46b0-8eb0-8da447c1f5a5 admin admin] PATH_INFO: `/v3/projects` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:41:59.723125 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cab94e85-d79a-46b0-8eb0-8da447c1f5a5 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:41:59.725056 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cab94e85-d79a-46b0-8eb0-8da447c1f5a5 admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:41:59.734541 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19adc927-6f1f-4062-8502-71c71c5fd75c None None] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:41:59.747357 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b115deb9-0fa7-43d1-a302-3d049f034be3 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f436f07939d4855a21b195bc7566244', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:41:59.748294 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b115deb9-0fa7-43d1-a302-3d049f034be3 service neutron] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:41:59.748605 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b115deb9-0fa7-43d1-a302-3d049f034be3 service neutron] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:41:59.748916 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b115deb9-0fa7-43d1-a302-3d049f034be3 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:41:59.759643 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b115deb9-0fa7-43d1-a302-3d049f034be3 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:41:59.761083 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b115deb9-0fa7-43d1-a302-3d049f034be3 service neutron] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:41:59.765272 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:41:59.768167 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1275/2532] 10.222.0.50 () {60 vars in 1153 bytes} [Tue Sep 13 16:41:59 2022] POST /identity/v3/projects => generated 401 bytes in 64 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 13 16:41:59.772403 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cab94e85-d79a-46b0-8eb0-8da447c1f5a5 admin admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:41:59.782750 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1258/2533] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 16:41:59 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 114 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 16:41:59.841615 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cab94e85-d79a-46b0-8eb0-8da447c1f5a5 admin admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:41:59.847828 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52dce046-2692-4fed-95d5-f209bffe88c4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:41:59.848512 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52dce046-2692-4fed-95d5-f209bffe88c4 admin admin] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:41:59.848816 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52dce046-2692-4fed-95d5-f209bffe88c4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:41:59.849564 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52dce046-2692-4fed-95d5-f209bffe88c4 admin admin] PATH_INFO: `/v3/users` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:41:59.849991 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52dce046-2692-4fed-95d5-f209bffe88c4 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:41:59.852232 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52dce046-2692-4fed-95d5-f209bffe88c4 admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:41:59.885181 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1276/2534] 10.222.0.50 () {60 vars in 1144 bytes} [Tue Sep 13 16:41:59 2022] POST /identity/v3/users => generated 301 bytes in 113 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 13 16:41:59.889477 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b115deb9-0fa7-43d1-a302-3d049f034be3 service neutron] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:41:59.904672 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b115deb9-0fa7-43d1-a302-3d049f034be3 service neutron] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:41:59.914667 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfd39818-4b48-4682-819e-04d89da530d7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:41:59.916031 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfd39818-4b48-4682-819e-04d89da530d7 admin admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:41:59.916423 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfd39818-4b48-4682-819e-04d89da530d7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:41:59.916937 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfd39818-4b48-4682-819e-04d89da530d7 admin admin] PATH_INFO: `/v3/roles` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:41:59.917676 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dfd39818-4b48-4682-819e-04d89da530d7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:41:59.919644 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dfd39818-4b48-4682-819e-04d89da530d7 admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:41:59.937513 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1259/2535] 10.222.0.50 () {58 vars in 1122 bytes} [Tue Sep 13 16:41:59 2022] GET /identity/v3/roles => generated 1420 bytes in 46 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 13 16:41:59.939668 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52dce046-2692-4fed-95d5-f209bffe88c4 admin admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:41:59.960572 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52dce046-2692-4fed-95d5-f209bffe88c4 admin admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:41:59.968395 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd7cae40-bb18-4c23-acac-22a8abc339d6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:41:59.968858 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd7cae40-bb18-4c23-acac-22a8abc339d6 admin admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:41:59.969253 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd7cae40-bb18-4c23-acac-22a8abc339d6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:41:59.969622 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd7cae40-bb18-4c23-acac-22a8abc339d6 admin admin] PATH_INFO: `/v3/roles` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:41:59.971733 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd7cae40-bb18-4c23-acac-22a8abc339d6 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:41:59.976316 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd7cae40-bb18-4c23-acac-22a8abc339d6 admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:42:00.014140 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1277/2536] 10.222.0.50 () {58 vars in 1122 bytes} [Tue Sep 13 16:41:59 2022] GET /identity/v3/roles => generated 1420 bytes in 76 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 16:42:00.019497 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfd39818-4b48-4682-819e-04d89da530d7 admin admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:42:00.031696 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfd39818-4b48-4682-819e-04d89da530d7 admin admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:42:00.045757 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07b16b68-5b63-41d3-a87c-9a7234eee301 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:42:00.047536 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07b16b68-5b63-41d3-a87c-9a7234eee301 admin admin] REQUEST_METHOD: `PUT` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:42:00.048395 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07b16b68-5b63-41d3-a87c-9a7234eee301 admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:42:00.048810 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07b16b68-5b63-41d3-a87c-9a7234eee301 admin admin] PATH_INFO: `/v3/projects/e4e1f8566af24fcaab73006738c35cce/users/ba7c69b25caa4529994e48856cb3a3c6/roles/15db4aaca3264ae29bde38903ba43471` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:42:00.069727 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07b16b68-5b63-41d3-a87c-9a7234eee301 admin admin] RBAC: Authorizing `identity:create_grant(project_id=e4e1f8566af24fcaab73006738c35cce, user_id=ba7c69b25caa4529994e48856cb3a3c6, role_id=15db4aaca3264ae29bde38903ba43471)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:42:00.073787 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07b16b68-5b63-41d3-a87c-9a7234eee301 admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:42:00.097863 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1260/2537] 10.222.0.50 () {60 vars in 1483 bytes} [Tue Sep 13 16:42:00 2022] PUT /identity/v3/projects/e4e1f8566af24fcaab73006738c35cce/users/ba7c69b25caa4529994e48856cb3a3c6/roles/15db4aaca3264ae29bde38903ba43471 => generated 0 bytes in 79 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 16:42:00.108864 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c2afaef-3a80-4290-8d71-d73a73fbf241 None None] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:42:00.109126 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c2afaef-3a80-4290-8d71-d73a73fbf241 None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:42:00.109409 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c2afaef-3a80-4290-8d71-d73a73fbf241 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:42:00.160135 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.auth.core [None req-0c2afaef-3a80-4290-8d71-d73a73fbf241 None None] MFA Rules not processed for user `ba7c69b25caa4529994e48856cb3a3c6`. Rule list: `[]` (Enabled: `True`). {{(pid=81437) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:42:00.161384 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-0c2afaef-3a80-4290-8d71-d73a73fbf241 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:42:00.167996 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=94,keystone:DELETE=2,keystone:INSERT=4 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:42:00.182174 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07b16b68-5b63-41d3-a87c-9a7234eee301 admin admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:42:00.184078 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-07b16b68-5b63-41d3-a87c-9a7234eee301 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:42:00.269237 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1278/2538] 10.222.0.50 () {58 vars in 958 bytes} [Tue Sep 13 16:42:00 2022] POST /identity/v3/auth/tokens => generated 3278 bytes in 161 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:42:00.273535 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c2afaef-3a80-4290-8d71-d73a73fbf241 None None] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:42:00.327206 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07b16b68-5b63-41d3-a87c-9a7234eee301 admin admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:42:00.333193 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f04992f7-6415-455f-8ce6-21653c690a6e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:42:00.333851 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f04992f7-6415-455f-8ce6-21653c690a6e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:42:00.334136 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f04992f7-6415-455f-8ce6-21653c690a6e admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:42:00.334447 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f04992f7-6415-455f-8ce6-21653c690a6e admin admin] PATH_INFO: `/v3/projects/463a112433c04cde96b50aa32d5d01e4` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:42:00.335603 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f04992f7-6415-455f-8ce6-21653c690a6e admin admin] RBAC: Authorizing `identity:delete_project(project_id=463a112433c04cde96b50aa32d5d01e4)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:42:00.337512 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f04992f7-6415-455f-8ce6-21653c690a6e admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:42:00.359142 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c2afaef-3a80-4290-8d71-d73a73fbf241 None None] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:42:00.364816 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a28a106-fa9a-42f1-8189-871857c320b6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:42:00.365426 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a28a106-fa9a-42f1-8189-871857c320b6 admin admin] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:42:00.365701 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a28a106-fa9a-42f1-8189-871857c320b6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:42:00.366101 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a28a106-fa9a-42f1-8189-871857c320b6 admin admin] PATH_INFO: `/v3/projects` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:42:00.366736 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a28a106-fa9a-42f1-8189-871857c320b6 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:42:00.368669 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a28a106-fa9a-42f1-8189-871857c320b6 admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:42:00.393345 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-f04992f7-6415-455f-8ce6-21653c690a6e admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '463a112433c04cde96b50aa32d5d01e4', 'request_id': 'req-f04992f7-6415-455f-8ce6-21653c690a6e'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:42:00.398760 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1279/2539] 10.222.0.50 () {60 vars in 1153 bytes} [Tue Sep 13 16:42:00 2022] POST /identity/v3/projects => generated 399 bytes in 126 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 16:42:00.399641 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a28a106-fa9a-42f1-8189-871857c320b6 admin admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:42:00.401641 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-3a28a106-fa9a-42f1-8189-871857c320b6 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:42:00.402013 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-f04992f7-6415-455f-8ce6-21653c690a6e admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '463a112433c04cde96b50aa32d5d01e4', 'request_id': 'req-f04992f7-6415-455f-8ce6-21653c690a6e'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:42:00.418886 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-f04992f7-6415-455f-8ce6-21653c690a6e admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '463a112433c04cde96b50aa32d5d01e4', 'request_id': 'req-f04992f7-6415-455f-8ce6-21653c690a6e'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:42:00.430513 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-f04992f7-6415-455f-8ce6-21653c690a6e admin admin] The token cache is being invalidate because project 463a112433c04cde96b50aa32d5d01e4 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81438) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 13 16:42:00.430807 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-f04992f7-6415-455f-8ce6-21653c690a6e admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:42:00.431540 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-f04992f7-6415-455f-8ce6-21653c690a6e admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:42:00.433319 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1261/2540] 10.222.0.50 () {58 vars in 1233 bytes} [Tue Sep 13 16:42:00 2022] DELETE /identity/v3/projects/463a112433c04cde96b50aa32d5d01e4 => generated 0 bytes in 251 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 13 16:42:00.434312 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f04992f7-6415-455f-8ce6-21653c690a6e admin admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:42:00.435876 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-f04992f7-6415-455f-8ce6-21653c690a6e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:42:00.474240 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a28a106-fa9a-42f1-8189-871857c320b6 admin admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:42:00.477978 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e72dfc02-24dd-4407-8681-3de997c6983a service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7fd6b5afd67b48f9b0f0597ccd4cf4df', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:42:00.478698 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e72dfc02-24dd-4407-8681-3de997c6983a service cinder] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:42:00.479010 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e72dfc02-24dd-4407-8681-3de997c6983a service cinder] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:42:00.479294 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e72dfc02-24dd-4407-8681-3de997c6983a service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:42:00.481063 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-e72dfc02-24dd-4407-8681-3de997c6983a service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:42:00.491710 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=91,keystone:INSERT=2,keystone:DELETE=7 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:42:00.507351 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f04992f7-6415-455f-8ce6-21653c690a6e admin admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:42:00.527859 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:42:00.542343 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e72dfc02-24dd-4407-8681-3de997c6983a service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:42:00.543904 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e72dfc02-24dd-4407-8681-3de997c6983a service cinder] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:42:00.558322 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1280/2541] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 16:42:00 2022] GET /identity/v3/auth/tokens => generated 3264 bytes in 159 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 16:42:00.559682 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e72dfc02-24dd-4407-8681-3de997c6983a service cinder] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:42:00.561701 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-e72dfc02-24dd-4407-8681-3de997c6983a service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:42:00.578828 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d88332e0-1123-49f5-9f64-18fe6c6341ab admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:42:00.580609 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d88332e0-1123-49f5-9f64-18fe6c6341ab admin admin] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:42:00.580896 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d88332e0-1123-49f5-9f64-18fe6c6341ab admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:42:00.581176 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d88332e0-1123-49f5-9f64-18fe6c6341ab admin admin] PATH_INFO: `/v3/users` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:42:00.581822 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d88332e0-1123-49f5-9f64-18fe6c6341ab admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:42:00.583884 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d88332e0-1123-49f5-9f64-18fe6c6341ab admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:42:00.608741 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1262/2542] 10.222.0.50 () {60 vars in 1144 bytes} [Tue Sep 13 16:42:00 2022] POST /identity/v3/users => generated 306 bytes in 175 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 16:42:00.613729 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d88332e0-1123-49f5-9f64-18fe6c6341ab admin admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:42:00.624316 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d88332e0-1123-49f5-9f64-18fe6c6341ab admin admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:42:00.629747 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02a14da7-566d-4fd7-be76-8fcdb891e6b2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:42:00.630312 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02a14da7-566d-4fd7-be76-8fcdb891e6b2 admin admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:42:00.630587 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02a14da7-566d-4fd7-be76-8fcdb891e6b2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:42:00.630866 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02a14da7-566d-4fd7-be76-8fcdb891e6b2 admin admin] PATH_INFO: `/v3/roles` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:42:00.631719 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02a14da7-566d-4fd7-be76-8fcdb891e6b2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:42:00.633331 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02a14da7-566d-4fd7-be76-8fcdb891e6b2 admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:42:00.639986 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e72dfc02-24dd-4407-8681-3de997c6983a service cinder] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:42:00.646415 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b010e319-d84e-46d5-9d3e-be7ca0aaa682 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:42:00.646997 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1263/2543] 10.222.0.50 () {58 vars in 1122 bytes} [Tue Sep 13 16:42:00 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 16:42:00.647205 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b010e319-d84e-46d5-9d3e-be7ca0aaa682 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:42:00.647623 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b010e319-d84e-46d5-9d3e-be7ca0aaa682 admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:42:00.647899 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b010e319-d84e-46d5-9d3e-be7ca0aaa682 admin admin] PATH_INFO: `/v3/users/0cd79a586b2f407b8ae6b19266c67036` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:42:00.649073 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b010e319-d84e-46d5-9d3e-be7ca0aaa682 admin admin] RBAC: Authorizing `identity:delete_user(user_id=0cd79a586b2f407b8ae6b19266c67036)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:42:00.651440 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b010e319-d84e-46d5-9d3e-be7ca0aaa682 admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:42:00.651587 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02a14da7-566d-4fd7-be76-8fcdb891e6b2 admin admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:42:00.665178 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02a14da7-566d-4fd7-be76-8fcdb891e6b2 admin admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:42:00.672573 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e9b19ff-46cd-440b-ba19-eaeb911be66f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:42:00.673235 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e9b19ff-46cd-440b-ba19-eaeb911be66f admin admin] REQUEST_METHOD: `PUT` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:42:00.673520 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e9b19ff-46cd-440b-ba19-eaeb911be66f admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:42:00.673798 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e9b19ff-46cd-440b-ba19-eaeb911be66f admin admin] PATH_INFO: `/v3/projects/86830c84bc9b44ae95573dfdff3f5bcf/users/6b7190f35fb94b3489ab34dd77a47c14/roles/031d4e1a23e34f71858fc4a6dfbd4ccb` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:42:00.693483 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e9b19ff-46cd-440b-ba19-eaeb911be66f admin admin] RBAC: Authorizing `identity:create_grant(project_id=86830c84bc9b44ae95573dfdff3f5bcf, user_id=6b7190f35fb94b3489ab34dd77a47c14, role_id=031d4e1a23e34f71858fc4a6dfbd4ccb)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:42:00.697657 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e9b19ff-46cd-440b-ba19-eaeb911be66f admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:42:00.718881 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1264/2544] 10.222.0.50 () {60 vars in 1483 bytes} [Tue Sep 13 16:42:00 2022] PUT /identity/v3/projects/86830c84bc9b44ae95573dfdff3f5bcf/users/6b7190f35fb94b3489ab34dd77a47c14/roles/031d4e1a23e34f71858fc4a6dfbd4ccb => generated 0 bytes in 68 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 13 16:42:00.723556 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-b010e319-d84e-46d5-9d3e-be7ca0aaa682 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '0cd79a586b2f407b8ae6b19266c67036'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:42:00.723886 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e9b19ff-46cd-440b-ba19-eaeb911be66f admin admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:42:00.732742 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-b010e319-d84e-46d5-9d3e-be7ca0aaa682 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '0cd79a586b2f407b8ae6b19266c67036'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:42:00.747937 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-b010e319-d84e-46d5-9d3e-be7ca0aaa682 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '0cd79a586b2f407b8ae6b19266c67036'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:42:00.748765 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-b010e319-d84e-46d5-9d3e-be7ca0aaa682 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '0cd79a586b2f407b8ae6b19266c67036'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:42:00.758341 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-b010e319-d84e-46d5-9d3e-be7ca0aaa682 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '0cd79a586b2f407b8ae6b19266c67036'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:42:00.765681 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1281/2545] 10.222.0.50 () {58 vars in 1224 bytes} [Tue Sep 13 16:42:00 2022] DELETE /identity/v3/users/0cd79a586b2f407b8ae6b19266c67036 => generated 0 bytes in 207 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 16:42:00.803584 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e9b19ff-46cd-440b-ba19-eaeb911be66f admin admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:42:00.888555 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-919cbce8-f6e8-4613-9a2d-29294431f4f8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:42:00.889366 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-919cbce8-f6e8-4613-9a2d-29294431f4f8 admin admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:42:00.889715 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-919cbce8-f6e8-4613-9a2d-29294431f4f8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:42:00.890060 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-919cbce8-f6e8-4613-9a2d-29294431f4f8 admin admin] PATH_INFO: `/v3/roles` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:42:00.891065 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-919cbce8-f6e8-4613-9a2d-29294431f4f8 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:42:00.893312 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-919cbce8-f6e8-4613-9a2d-29294431f4f8 admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:42:00.896447 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:42:00.912991 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1265/2546] 10.222.0.50 () {58 vars in 1122 bytes} [Tue Sep 13 16:42:00 2022] GET /identity/v3/roles => generated 1420 bytes in 188 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 13 16:42:00.914721 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b010e319-d84e-46d5-9d3e-be7ca0aaa682 admin admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:42:00.916497 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-b010e319-d84e-46d5-9d3e-be7ca0aaa682 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:42:00.932952 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b010e319-d84e-46d5-9d3e-be7ca0aaa682 admin admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:42:00.939860 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5f13eca-da43-41db-b532-1ae275103841 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:42:00.940577 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5f13eca-da43-41db-b532-1ae275103841 admin admin] REQUEST_METHOD: `PUT` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:42:00.940900 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5f13eca-da43-41db-b532-1ae275103841 admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:42:00.941211 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5f13eca-da43-41db-b532-1ae275103841 admin admin] PATH_INFO: `/v3/projects/86830c84bc9b44ae95573dfdff3f5bcf/users/6b7190f35fb94b3489ab34dd77a47c14/roles/031d4e1a23e34f71858fc4a6dfbd4ccb` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:42:00.956877 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5f13eca-da43-41db-b532-1ae275103841 admin admin] RBAC: Authorizing `identity:create_grant(project_id=86830c84bc9b44ae95573dfdff3f5bcf, user_id=6b7190f35fb94b3489ab34dd77a47c14, role_id=031d4e1a23e34f71858fc4a6dfbd4ccb)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:42:00.963793 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5f13eca-da43-41db-b532-1ae275103841 admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:42:00.989588 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1282/2547] 10.222.0.50 () {60 vars in 1483 bytes} [Tue Sep 13 16:42:00 2022] PUT /identity/v3/projects/86830c84bc9b44ae95573dfdff3f5bcf/users/6b7190f35fb94b3489ab34dd77a47c14/roles/031d4e1a23e34f71858fc4a6dfbd4ccb => generated 0 bytes in 76 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 16:42:01.001215 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54d8724a-cb43-4189-9271-17790dbabd0c None None] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:42:01.001543 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54d8724a-cb43-4189-9271-17790dbabd0c None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:42:01.001798 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54d8724a-cb43-4189-9271-17790dbabd0c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:42:01.052667 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.auth.core [None req-54d8724a-cb43-4189-9271-17790dbabd0c None None] MFA Rules not processed for user `6b7190f35fb94b3489ab34dd77a47c14`. Rule list: `[]` (Enabled: `True`). {{(pid=81438) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:42:01.054143 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-54d8724a-cb43-4189-9271-17790dbabd0c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:42:01.143807 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5f13eca-da43-41db-b532-1ae275103841 admin admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:42:01.149261 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-a5f13eca-da43-41db-b532-1ae275103841 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:42:01.179220 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1266/2548] 10.222.0.50 () {58 vars in 958 bytes} [Tue Sep 13 16:42:00 2022] POST /identity/v3/auth/tokens => generated 3343 bytes in 177 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:42:01.191706 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54d8724a-cb43-4189-9271-17790dbabd0c None None] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:42:01.193145 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-54d8724a-cb43-4189-9271-17790dbabd0c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:42:01.274353 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=88,keystone:DELETE=10,keystone:INSERT=2 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:42:01.292981 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5f13eca-da43-41db-b532-1ae275103841 admin admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:42:01.300106 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e3af95b-8bab-483e-8def-90750a974c43 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:42:01.300780 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e3af95b-8bab-483e-8def-90750a974c43 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:42:01.301235 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e3af95b-8bab-483e-8def-90750a974c43 admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:42:01.301955 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e3af95b-8bab-483e-8def-90750a974c43 admin admin] PATH_INFO: `/v3/projects/dc73199ebe994b159e5877db5b604b3e` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:42:01.304203 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e3af95b-8bab-483e-8def-90750a974c43 admin admin] RBAC: Authorizing `identity:delete_project(project_id=dc73199ebe994b159e5877db5b604b3e)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:42:01.306163 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54d8724a-cb43-4189-9271-17790dbabd0c None None] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:42:01.307931 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e3af95b-8bab-483e-8def-90750a974c43 admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:42:01.311709 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-590b343d-34eb-4ca0-ad86-84349cc9f01b service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7fd6b5afd67b48f9b0f0597ccd4cf4df', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:42:01.312446 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-590b343d-34eb-4ca0-ad86-84349cc9f01b service cinder] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:42:01.313797 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-590b343d-34eb-4ca0-ad86-84349cc9f01b service cinder] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:42:01.314273 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-590b343d-34eb-4ca0-ad86-84349cc9f01b service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:42:01.329627 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-590b343d-34eb-4ca0-ad86-84349cc9f01b service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:42:01.329627 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-590b343d-34eb-4ca0-ad86-84349cc9f01b service cinder] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:42:01.340386 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1267/2549] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 16:42:01 2022] GET /identity/v3/auth/tokens => generated 3343 bytes in 152 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 16:42:01.410143 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-2e3af95b-8bab-483e-8def-90750a974c43 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'dc73199ebe994b159e5877db5b604b3e', 'request_id': 'req-2e3af95b-8bab-483e-8def-90750a974c43'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:42:01.410143 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-2e3af95b-8bab-483e-8def-90750a974c43 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'dc73199ebe994b159e5877db5b604b3e', 'request_id': 'req-2e3af95b-8bab-483e-8def-90750a974c43'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:42:01.424987 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-2e3af95b-8bab-483e-8def-90750a974c43 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'dc73199ebe994b159e5877db5b604b3e', 'request_id': 'req-2e3af95b-8bab-483e-8def-90750a974c43'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:42:01.444403 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-2e3af95b-8bab-483e-8def-90750a974c43 admin admin] The token cache is being invalidate because project dc73199ebe994b159e5877db5b604b3e was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81437) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 13 16:42:01.444654 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-2e3af95b-8bab-483e-8def-90750a974c43 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:42:01.447183 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-2e3af95b-8bab-483e-8def-90750a974c43 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:42:01.458047 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1283/2550] 10.222.0.50 () {58 vars in 1233 bytes} [Tue Sep 13 16:42:01 2022] DELETE /identity/v3/projects/dc73199ebe994b159e5877db5b604b3e => generated 0 bytes in 308 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 16:42:01.458134 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f25cdf80-08fb-4d0d-9efc-d930c7c6e7cf None None] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:42:01.458134 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f25cdf80-08fb-4d0d-9efc-d930c7c6e7cf None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:42:01.458134 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f25cdf80-08fb-4d0d-9efc-d930c7c6e7cf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:42:01.465262 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e3af95b-8bab-483e-8def-90750a974c43 admin admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:42:01.466675 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-2e3af95b-8bab-483e-8def-90750a974c43 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:42:01.479273 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.auth.core [None req-f25cdf80-08fb-4d0d-9efc-d930c7c6e7cf None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81438) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:42:01.480452 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-f25cdf80-08fb-4d0d-9efc-d930c7c6e7cf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:42:01.542378 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e3af95b-8bab-483e-8def-90750a974c43 admin admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:42:01.549669 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3958af5-8df6-4077-96a2-27fa8b9f0f06 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7fd6b5afd67b48f9b0f0597ccd4cf4df', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:42:01.550971 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3958af5-8df6-4077-96a2-27fa8b9f0f06 service cinder] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:42:01.551817 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3958af5-8df6-4077-96a2-27fa8b9f0f06 service cinder] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:42:01.552265 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3958af5-8df6-4077-96a2-27fa8b9f0f06 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:42:01.554688 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-f3958af5-8df6-4077-96a2-27fa8b9f0f06 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:42:01.582947 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1268/2551] 10.222.0.50 () {58 vars in 958 bytes} [Tue Sep 13 16:42:01 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 124 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:42:01.588170 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d27df3f9-d89b-4b2e-b960-e223d547ce8f None None] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:42:01.588526 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d27df3f9-d89b-4b2e-b960-e223d547ce8f None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:42:01.588816 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d27df3f9-d89b-4b2e-b960-e223d547ce8f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:42:01.621086 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.auth.core [None req-d27df3f9-d89b-4b2e-b960-e223d547ce8f None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81438) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:42:01.622472 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-d27df3f9-d89b-4b2e-b960-e223d547ce8f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:42:01.637660 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1269/2552] 10.222.0.50 () {58 vars in 958 bytes} [Tue Sep 13 16:42:01 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 51 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 13 16:42:01.642483 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d27df3f9-d89b-4b2e-b960-e223d547ce8f None None] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:42:01.653970 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d27df3f9-d89b-4b2e-b960-e223d547ce8f None None] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:42:01.660662 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ef33b91-df12-4f27-9c12-7c87b1f446ba admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:42:01.662008 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ef33b91-df12-4f27-9c12-7c87b1f446ba admin admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:42:01.662779 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ef33b91-df12-4f27-9c12-7c87b1f446ba admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:42:01.663180 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ef33b91-df12-4f27-9c12-7c87b1f446ba admin admin] PATH_INFO: `/v3/domains` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:42:01.664010 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ef33b91-df12-4f27-9c12-7c87b1f446ba admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81438) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 16:42:01.664333 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ef33b91-df12-4f27-9c12-7c87b1f446ba admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:42:01.664551 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3958af5-8df6-4077-96a2-27fa8b9f0f06 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:42:01.666158 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3958af5-8df6-4077-96a2-27fa8b9f0f06 service cinder] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:42:01.666558 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ef33b91-df12-4f27-9c12-7c87b1f446ba admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:42:01.679912 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1284/2553] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 16:42:01 2022] GET /identity/v3/auth/tokens => generated 3278 bytes in 215 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 13 16:42:01.684011 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1270/2554] 10.222.0.50 () {58 vars in 1153 bytes} [Tue Sep 13 16:42:01 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 42 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 16:42:01.689136 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3958af5-8df6-4077-96a2-27fa8b9f0f06 service cinder] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:42:01.700178 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3958af5-8df6-4077-96a2-27fa8b9f0f06 service cinder] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:42:01.706097 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2802f375-102c-485d-860f-fed247af5701 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:42:01.706843 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2802f375-102c-485d-860f-fed247af5701 admin admin] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:42:01.707234 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2802f375-102c-485d-860f-fed247af5701 admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:42:01.707513 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2802f375-102c-485d-860f-fed247af5701 admin admin] PATH_INFO: `/v3/projects` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:42:01.708155 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2802f375-102c-485d-860f-fed247af5701 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:42:01.710075 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2802f375-102c-485d-860f-fed247af5701 admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:42:01.745697 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1285/2555] 10.222.0.50 () {60 vars in 1153 bytes} [Tue Sep 13 16:42:01 2022] POST /identity/v3/projects => generated 387 bytes in 58 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 16:42:01.750738 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ef33b91-df12-4f27-9c12-7c87b1f446ba admin admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:42:01.762253 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ef33b91-df12-4f27-9c12-7c87b1f446ba admin admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:42:01.767392 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=100 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:42:01.857549 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88e477a6-3297-47e2-810f-4129c6c86027 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:42:01.858184 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88e477a6-3297-47e2-810f-4129c6c86027 admin admin] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:42:01.858467 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88e477a6-3297-47e2-810f-4129c6c86027 admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:42:01.858744 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88e477a6-3297-47e2-810f-4129c6c86027 admin admin] PATH_INFO: `/v3/users` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:42:01.859489 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88e477a6-3297-47e2-810f-4129c6c86027 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:42:01.862410 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88e477a6-3297-47e2-810f-4129c6c86027 admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:42:01.896786 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1271/2556] 10.222.0.50 () {60 vars in 1144 bytes} [Tue Sep 13 16:42:01 2022] POST /identity/v3/users => generated 294 bytes in 147 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 13 16:42:01.900921 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2802f375-102c-485d-860f-fed247af5701 admin admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:42:01.916704 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2802f375-102c-485d-860f-fed247af5701 admin admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:42:01.923908 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d50b4cab-69e9-4bb7-99cf-1f26623bb447 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:42:01.925239 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d50b4cab-69e9-4bb7-99cf-1f26623bb447 admin admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:42:01.925729 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d50b4cab-69e9-4bb7-99cf-1f26623bb447 admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:42:01.926344 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d50b4cab-69e9-4bb7-99cf-1f26623bb447 admin admin] PATH_INFO: `/v3/roles` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:42:01.927512 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d50b4cab-69e9-4bb7-99cf-1f26623bb447 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:42:01.929561 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d50b4cab-69e9-4bb7-99cf-1f26623bb447 admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:42:01.948743 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1286/2557] 10.222.0.50 () {58 vars in 1122 bytes} [Tue Sep 13 16:42:01 2022] GET /identity/v3/roles => generated 1420 bytes in 48 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 13 16:42:01.964421 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88e477a6-3297-47e2-810f-4129c6c86027 admin admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:42:01.986392 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88e477a6-3297-47e2-810f-4129c6c86027 admin admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:42:02.000588 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f8ae13d-a70e-49dc-91ab-77638126eb1e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:42:02.001310 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f8ae13d-a70e-49dc-91ab-77638126eb1e admin admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:42:02.001593 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f8ae13d-a70e-49dc-91ab-77638126eb1e admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:42:02.001898 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f8ae13d-a70e-49dc-91ab-77638126eb1e admin admin] PATH_INFO: `/v3/roles` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:42:02.003051 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f8ae13d-a70e-49dc-91ab-77638126eb1e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:42:02.007729 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f8ae13d-a70e-49dc-91ab-77638126eb1e admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:42:02.020925 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1272/2558] 10.222.0.50 () {58 vars in 1122 bytes} [Tue Sep 13 16:42:01 2022] GET /identity/v3/roles => generated 1420 bytes in 60 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 16:42:02.025248 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d50b4cab-69e9-4bb7-99cf-1f26623bb447 admin admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:42:02.034259 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:42:02.037145 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d50b4cab-69e9-4bb7-99cf-1f26623bb447 admin admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:42:02.048993 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37d4510f-a639-49a8-bcf8-67d26d33d6b6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:42:02.050074 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37d4510f-a639-49a8-bcf8-67d26d33d6b6 admin admin] REQUEST_METHOD: `PUT` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:42:02.050364 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37d4510f-a639-49a8-bcf8-67d26d33d6b6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:42:02.050686 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37d4510f-a639-49a8-bcf8-67d26d33d6b6 admin admin] PATH_INFO: `/v3/projects/f49c227e708f477fb8e16ff80067a085/users/9abc8c189dae49fc81ccce002b331eb9/roles/15db4aaca3264ae29bde38903ba43471` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:42:02.077625 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37d4510f-a639-49a8-bcf8-67d26d33d6b6 admin admin] RBAC: Authorizing `identity:create_grant(project_id=f49c227e708f477fb8e16ff80067a085, user_id=9abc8c189dae49fc81ccce002b331eb9, role_id=15db4aaca3264ae29bde38903ba43471)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:42:02.090996 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37d4510f-a639-49a8-bcf8-67d26d33d6b6 admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:42:02.119028 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1287/2559] 10.222.0.50 () {60 vars in 1483 bytes} [Tue Sep 13 16:42:02 2022] PUT /identity/v3/projects/f49c227e708f477fb8e16ff80067a085/users/9abc8c189dae49fc81ccce002b331eb9/roles/15db4aaca3264ae29bde38903ba43471 => generated 0 bytes in 94 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 13 16:42:02.134687 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecfc89e6-1f47-4629-8f90-ad7343df5bbe None None] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:42:02.135232 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecfc89e6-1f47-4629-8f90-ad7343df5bbe None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:42:02.135876 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecfc89e6-1f47-4629-8f90-ad7343df5bbe None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:42:02.187800 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.auth.core [None req-ecfc89e6-1f47-4629-8f90-ad7343df5bbe None None] MFA Rules not processed for user `9abc8c189dae49fc81ccce002b331eb9`. Rule list: `[]` (Enabled: `True`). {{(pid=81438) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:42:02.190434 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-ecfc89e6-1f47-4629-8f90-ad7343df5bbe None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:42:02.266376 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1273/2560] 10.222.0.50 () {58 vars in 958 bytes} [Tue Sep 13 16:42:02 2022] POST /identity/v3/auth/tokens => generated 3264 bytes in 134 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:42:02.271642 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37d4510f-a639-49a8-bcf8-67d26d33d6b6 admin admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:42:02.347335 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37d4510f-a639-49a8-bcf8-67d26d33d6b6 admin admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:42:02.367417 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15d8189e-0090-4810-9344-73c23abb685d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:42:02.368049 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15d8189e-0090-4810-9344-73c23abb685d admin admin] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:42:02.368332 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15d8189e-0090-4810-9344-73c23abb685d admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:42:02.368721 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15d8189e-0090-4810-9344-73c23abb685d admin admin] PATH_INFO: `/v3/projects` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:42:02.369262 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15d8189e-0090-4810-9344-73c23abb685d admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:42:02.371278 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15d8189e-0090-4810-9344-73c23abb685d admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:42:02.408228 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1288/2561] 10.222.0.50 () {60 vars in 1153 bytes} [Tue Sep 13 16:42:02 2022] POST /identity/v3/projects => generated 385 bytes in 137 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 16:42:02.412480 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecfc89e6-1f47-4629-8f90-ad7343df5bbe None None] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:42:02.425027 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecfc89e6-1f47-4629-8f90-ad7343df5bbe None None] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:42:02.486659 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e541fbe-9c9d-47f5-8284-44075e3f2d89 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:42:02.487303 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e541fbe-9c9d-47f5-8284-44075e3f2d89 admin admin] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:42:02.487583 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e541fbe-9c9d-47f5-8284-44075e3f2d89 admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:42:02.487972 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e541fbe-9c9d-47f5-8284-44075e3f2d89 admin admin] PATH_INFO: `/v3/users` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:42:02.488631 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e541fbe-9c9d-47f5-8284-44075e3f2d89 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:42:02.490550 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e541fbe-9c9d-47f5-8284-44075e3f2d89 admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:42:02.517431 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1274/2562] 10.222.0.50 () {60 vars in 1144 bytes} [Tue Sep 13 16:42:02 2022] POST /identity/v3/users => generated 299 bytes in 102 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 16:42:02.520195 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15d8189e-0090-4810-9344-73c23abb685d admin admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:42:02.535357 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15d8189e-0090-4810-9344-73c23abb685d admin admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:42:02.541938 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe5e4aa2-c358-4e69-abc5-8bbaa16b1281 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:42:02.542559 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe5e4aa2-c358-4e69-abc5-8bbaa16b1281 admin admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:42:02.542837 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe5e4aa2-c358-4e69-abc5-8bbaa16b1281 admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:42:02.543563 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe5e4aa2-c358-4e69-abc5-8bbaa16b1281 admin admin] PATH_INFO: `/v3/roles` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:42:02.543942 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe5e4aa2-c358-4e69-abc5-8bbaa16b1281 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:42:02.545562 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe5e4aa2-c358-4e69-abc5-8bbaa16b1281 admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:42:02.557214 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1289/2563] 10.222.0.50 () {58 vars in 1122 bytes} [Tue Sep 13 16:42:02 2022] GET /identity/v3/roles => generated 1420 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 16:42:02.561435 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e541fbe-9c9d-47f5-8284-44075e3f2d89 admin admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:42:02.567953 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=94,keystone:INSERT=6 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:42:02.571934 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e541fbe-9c9d-47f5-8284-44075e3f2d89 admin admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:42:02.577795 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0d04d3a-b29c-48fa-90b8-763409f3eb29 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:42:02.578538 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0d04d3a-b29c-48fa-90b8-763409f3eb29 admin admin] REQUEST_METHOD: `PUT` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:42:02.578945 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0d04d3a-b29c-48fa-90b8-763409f3eb29 admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:42:02.579366 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0d04d3a-b29c-48fa-90b8-763409f3eb29 admin admin] PATH_INFO: `/v3/projects/b8a301bb55ff4b5f81701e0d843614f9/users/180031cda72f4862974b99219f71a9d5/roles/031d4e1a23e34f71858fc4a6dfbd4ccb` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:42:02.595192 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0d04d3a-b29c-48fa-90b8-763409f3eb29 admin admin] RBAC: Authorizing `identity:create_grant(project_id=b8a301bb55ff4b5f81701e0d843614f9, user_id=180031cda72f4862974b99219f71a9d5, role_id=031d4e1a23e34f71858fc4a6dfbd4ccb)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:42:02.598272 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0d04d3a-b29c-48fa-90b8-763409f3eb29 admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:42:02.613158 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1275/2564] 10.222.0.50 () {60 vars in 1483 bytes} [Tue Sep 13 16:42:02 2022] PUT /identity/v3/projects/b8a301bb55ff4b5f81701e0d843614f9/users/180031cda72f4862974b99219f71a9d5/roles/031d4e1a23e34f71858fc4a6dfbd4ccb => generated 0 bytes in 52 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 16:42:02.617371 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe5e4aa2-c358-4e69-abc5-8bbaa16b1281 admin admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:42:02.628633 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe5e4aa2-c358-4e69-abc5-8bbaa16b1281 admin admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:42:02.700467 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c720c515-ccef-4668-8c3d-ec1d7b340ead admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:42:02.701082 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c720c515-ccef-4668-8c3d-ec1d7b340ead admin admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:42:02.701360 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c720c515-ccef-4668-8c3d-ec1d7b340ead admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:42:02.701629 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c720c515-ccef-4668-8c3d-ec1d7b340ead admin admin] PATH_INFO: `/v3/roles` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:42:02.702270 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c720c515-ccef-4668-8c3d-ec1d7b340ead admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:42:02.703972 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c720c515-ccef-4668-8c3d-ec1d7b340ead admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:42:02.715881 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1290/2565] 10.222.0.50 () {58 vars in 1122 bytes} [Tue Sep 13 16:42:02 2022] GET /identity/v3/roles => generated 1420 bytes in 99 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 16:42:02.720441 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0d04d3a-b29c-48fa-90b8-763409f3eb29 admin admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:42:02.737267 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0d04d3a-b29c-48fa-90b8-763409f3eb29 admin admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:42:02.747019 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38457779-f5b2-4c58-acfb-cfaf5ca93eeb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:42:02.747775 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38457779-f5b2-4c58-acfb-cfaf5ca93eeb admin admin] REQUEST_METHOD: `PUT` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:42:02.748126 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38457779-f5b2-4c58-acfb-cfaf5ca93eeb admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:42:02.749014 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38457779-f5b2-4c58-acfb-cfaf5ca93eeb admin admin] PATH_INFO: `/v3/projects/b8a301bb55ff4b5f81701e0d843614f9/users/180031cda72f4862974b99219f71a9d5/roles/031d4e1a23e34f71858fc4a6dfbd4ccb` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:42:02.751186 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38457779-f5b2-4c58-acfb-cfaf5ca93eeb admin admin] RBAC: Authorizing `identity:create_grant(project_id=b8a301bb55ff4b5f81701e0d843614f9, user_id=180031cda72f4862974b99219f71a9d5, role_id=031d4e1a23e34f71858fc4a6dfbd4ccb)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:42:02.753954 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38457779-f5b2-4c58-acfb-cfaf5ca93eeb admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:42:02.765370 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1276/2566] 10.222.0.50 () {60 vars in 1483 bytes} [Tue Sep 13 16:42:02 2022] PUT /identity/v3/projects/b8a301bb55ff4b5f81701e0d843614f9/users/180031cda72f4862974b99219f71a9d5/roles/031d4e1a23e34f71858fc4a6dfbd4ccb => generated 0 bytes in 46 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 16:42:02.777852 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abf5bc83-b40e-48f7-8c08-3018ad1114ad None None] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:42:02.777983 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abf5bc83-b40e-48f7-8c08-3018ad1114ad None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:42:02.778181 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abf5bc83-b40e-48f7-8c08-3018ad1114ad None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:42:02.809649 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.auth.core [None req-abf5bc83-b40e-48f7-8c08-3018ad1114ad None None] MFA Rules not processed for user `180031cda72f4862974b99219f71a9d5`. Rule list: `[]` (Enabled: `True`). {{(pid=81437) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:42:02.810992 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-abf5bc83-b40e-48f7-8c08-3018ad1114ad None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:42:02.868530 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:42:02.904434 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1291/2567] 10.222.0.50 () {58 vars in 958 bytes} [Tue Sep 13 16:42:02 2022] POST /identity/v3/auth/tokens => generated 3329 bytes in 128 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:42:02.919122 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38457779-f5b2-4c58-acfb-cfaf5ca93eeb admin admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:42:02.921057 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-38457779-f5b2-4c58-acfb-cfaf5ca93eeb admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:42:03.017443 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38457779-f5b2-4c58-acfb-cfaf5ca93eeb admin admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:42:03.021445 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58075d4c-796b-44c4-aeb8-adf90fe758d5 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7fd6b5afd67b48f9b0f0597ccd4cf4df', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:42:03.022071 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58075d4c-796b-44c4-aeb8-adf90fe758d5 service cinder] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:42:03.022353 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58075d4c-796b-44c4-aeb8-adf90fe758d5 service cinder] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:42:03.022630 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58075d4c-796b-44c4-aeb8-adf90fe758d5 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:42:03.033259 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58075d4c-796b-44c4-aeb8-adf90fe758d5 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:42:03.034711 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58075d4c-796b-44c4-aeb8-adf90fe758d5 service cinder] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:42:03.055069 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1277/2568] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 16:42:02 2022] GET /identity/v3/auth/tokens => generated 3329 bytes in 137 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 16:42:03.135091 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abf5bc83-b40e-48f7-8c08-3018ad1114ad None None] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:42:03.227615 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abf5bc83-b40e-48f7-8c08-3018ad1114ad None None] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:42:03.232822 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36663dc8-7e3f-4cf4-ab0f-72ec8a3b1dd8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:42:03.233469 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36663dc8-7e3f-4cf4-ab0f-72ec8a3b1dd8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:42:03.233749 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36663dc8-7e3f-4cf4-ab0f-72ec8a3b1dd8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:42:03.234028 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36663dc8-7e3f-4cf4-ab0f-72ec8a3b1dd8 admin admin] PATH_INFO: `/v3/users/9abc8c189dae49fc81ccce002b331eb9` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:42:03.235523 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36663dc8-7e3f-4cf4-ab0f-72ec8a3b1dd8 admin admin] RBAC: Authorizing `identity:delete_user(user_id=9abc8c189dae49fc81ccce002b331eb9)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:42:03.240983 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36663dc8-7e3f-4cf4-ab0f-72ec8a3b1dd8 admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:42:03.282473 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58075d4c-796b-44c4-aeb8-adf90fe758d5 service cinder] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:42:03.284688 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-58075d4c-796b-44c4-aeb8-adf90fe758d5 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:42:03.304549 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58075d4c-796b-44c4-aeb8-adf90fe758d5 service cinder] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:42:03.318499 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7f134e8-9b82-4c90-8ff0-91a529713a0f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:42:03.319575 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7f134e8-9b82-4c90-8ff0-91a529713a0f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:42:03.320011 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7f134e8-9b82-4c90-8ff0-91a529713a0f admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:42:03.320482 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7f134e8-9b82-4c90-8ff0-91a529713a0f admin admin] PATH_INFO: `/v3/users/d29755d7854441c8a5c85aa16c7a05f5` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:42:03.322465 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7f134e8-9b82-4c90-8ff0-91a529713a0f admin admin] RBAC: Authorizing `identity:delete_user(user_id=d29755d7854441c8a5c85aa16c7a05f5)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:42:03.325951 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-36663dc8-7e3f-4cf4-ab0f-72ec8a3b1dd8 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '9abc8c189dae49fc81ccce002b331eb9'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:42:03.329623 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7f134e8-9b82-4c90-8ff0-91a529713a0f admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:42:03.337568 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-36663dc8-7e3f-4cf4-ab0f-72ec8a3b1dd8 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '9abc8c189dae49fc81ccce002b331eb9'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:42:03.353813 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-36663dc8-7e3f-4cf4-ab0f-72ec8a3b1dd8 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '9abc8c189dae49fc81ccce002b331eb9'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:42:03.354647 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-36663dc8-7e3f-4cf4-ab0f-72ec8a3b1dd8 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '9abc8c189dae49fc81ccce002b331eb9'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:42:03.363220 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-36663dc8-7e3f-4cf4-ab0f-72ec8a3b1dd8 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '9abc8c189dae49fc81ccce002b331eb9'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:42:03.371354 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1292/2569] 10.222.0.50 () {58 vars in 1224 bytes} [Tue Sep 13 16:42:03 2022] DELETE /identity/v3/users/9abc8c189dae49fc81ccce002b331eb9 => generated 0 bytes in 237 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 16:42:03.377122 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19978d2a-9efa-4af2-8832-4e6fad31ff4c None None] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:42:03.379468 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19978d2a-9efa-4af2-8832-4e6fad31ff4c None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:42:03.379962 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19978d2a-9efa-4af2-8832-4e6fad31ff4c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:42:03.412238 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-a7f134e8-9b82-4c90-8ff0-91a529713a0f admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'd29755d7854441c8a5c85aa16c7a05f5'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:42:03.424371 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-a7f134e8-9b82-4c90-8ff0-91a529713a0f admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'd29755d7854441c8a5c85aa16c7a05f5'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:42:03.434178 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.auth.core [None req-19978d2a-9efa-4af2-8832-4e6fad31ff4c None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81437) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:42:03.437146 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-19978d2a-9efa-4af2-8832-4e6fad31ff4c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:42:03.442531 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-a7f134e8-9b82-4c90-8ff0-91a529713a0f admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'd29755d7854441c8a5c85aa16c7a05f5'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:42:03.443735 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-a7f134e8-9b82-4c90-8ff0-91a529713a0f admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'd29755d7854441c8a5c85aa16c7a05f5'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:42:03.455398 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-a7f134e8-9b82-4c90-8ff0-91a529713a0f admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'd29755d7854441c8a5c85aa16c7a05f5'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:42:03.466281 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1278/2570] 10.222.0.50 () {58 vars in 1224 bytes} [Tue Sep 13 16:42:03 2022] DELETE /identity/v3/users/d29755d7854441c8a5c85aa16c7a05f5 => generated 0 bytes in 185 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 13 16:42:03.471492 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b43e425-e8df-47f0-b9f6-01e31d28b94c None None] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:42:03.471777 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b43e425-e8df-47f0-b9f6-01e31d28b94c None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:42:03.472061 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b43e425-e8df-47f0-b9f6-01e31d28b94c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:42:03.492659 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.auth.core [None req-0b43e425-e8df-47f0-b9f6-01e31d28b94c None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81438) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:42:03.494064 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-0b43e425-e8df-47f0-b9f6-01e31d28b94c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:42:03.500442 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:42:03.517239 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1293/2571] 10.222.0.50 () {58 vars in 958 bytes} [Tue Sep 13 16:42:03 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 142 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 13 16:42:03.537333 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19978d2a-9efa-4af2-8832-4e6fad31ff4c None None] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:42:03.539128 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-19978d2a-9efa-4af2-8832-4e6fad31ff4c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:42:03.546152 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=87,keystone:INSERT=3,keystone:DELETE=10 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:42:03.576561 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1279/2572] 10.222.0.50 () {58 vars in 958 bytes} [Tue Sep 13 16:42:03 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:42:03.587786 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b43e425-e8df-47f0-b9f6-01e31d28b94c None None] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:42:03.589368 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-0b43e425-e8df-47f0-b9f6-01e31d28b94c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:42:03.622664 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19978d2a-9efa-4af2-8832-4e6fad31ff4c None None] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:42:03.657852 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b43e425-e8df-47f0-b9f6-01e31d28b94c None None] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:42:03.661865 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6406242b-5856-4670-bdb2-4bda3d34dfba service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f436f07939d4855a21b195bc7566244', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:42:03.662473 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6406242b-5856-4670-bdb2-4bda3d34dfba service neutron] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:42:03.662750 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6406242b-5856-4670-bdb2-4bda3d34dfba service neutron] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:42:03.663072 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6406242b-5856-4670-bdb2-4bda3d34dfba service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:42:03.675966 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6406242b-5856-4670-bdb2-4bda3d34dfba service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:42:03.677390 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6406242b-5856-4670-bdb2-4bda3d34dfba service neutron] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:42:03.678404 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24449403-f9f3-401e-8625-7b589c985608 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f436f07939d4855a21b195bc7566244', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:42:03.679109 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24449403-f9f3-401e-8625-7b589c985608 service neutron] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:42:03.679392 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24449403-f9f3-401e-8625-7b589c985608 service neutron] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:42:03.679669 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24449403-f9f3-401e-8625-7b589c985608 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:42:03.689864 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24449403-f9f3-401e-8625-7b589c985608 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:42:03.690584 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1280/2573] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 16:42:03 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 103 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 13 16:42:03.691380 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24449403-f9f3-401e-8625-7b589c985608 service neutron] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:42:03.701587 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1294/2574] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 16:42:03 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 165 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 13 16:42:04.111520 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6406242b-5856-4670-bdb2-4bda3d34dfba service neutron] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:42:04.113940 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-6406242b-5856-4670-bdb2-4bda3d34dfba service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:42:04.134983 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6406242b-5856-4670-bdb2-4bda3d34dfba service neutron] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:42:04.145619 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5f8fa0f-10eb-47af-8dc0-90197aa7cd83 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:42:04.148287 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5f8fa0f-10eb-47af-8dc0-90197aa7cd83 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:42:04.149736 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5f8fa0f-10eb-47af-8dc0-90197aa7cd83 admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:42:04.151419 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5f8fa0f-10eb-47af-8dc0-90197aa7cd83 admin admin] PATH_INFO: `/v3/projects/f49c227e708f477fb8e16ff80067a085` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:42:04.153116 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5f8fa0f-10eb-47af-8dc0-90197aa7cd83 admin admin] RBAC: Authorizing `identity:delete_project(project_id=f49c227e708f477fb8e16ff80067a085)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:42:04.155434 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5f8fa0f-10eb-47af-8dc0-90197aa7cd83 admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:42:04.181398 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24449403-f9f3-401e-8625-7b589c985608 service neutron] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:42:04.183430 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-24449403-f9f3-401e-8625-7b589c985608 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:42:04.206329 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24449403-f9f3-401e-8625-7b589c985608 service neutron] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:42:04.221091 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-d5f8fa0f-10eb-47af-8dc0-90197aa7cd83 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'f49c227e708f477fb8e16ff80067a085', 'request_id': 'req-d5f8fa0f-10eb-47af-8dc0-90197aa7cd83'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:42:04.230942 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-d5f8fa0f-10eb-47af-8dc0-90197aa7cd83 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'f49c227e708f477fb8e16ff80067a085', 'request_id': 'req-d5f8fa0f-10eb-47af-8dc0-90197aa7cd83'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:42:04.238550 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-d5f8fa0f-10eb-47af-8dc0-90197aa7cd83 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'f49c227e708f477fb8e16ff80067a085', 'request_id': 'req-d5f8fa0f-10eb-47af-8dc0-90197aa7cd83'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:42:04.251635 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-d5f8fa0f-10eb-47af-8dc0-90197aa7cd83 admin admin] The token cache is being invalidate because project f49c227e708f477fb8e16ff80067a085 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81438) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 13 16:42:04.252084 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-d5f8fa0f-10eb-47af-8dc0-90197aa7cd83 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:42:04.252941 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-d5f8fa0f-10eb-47af-8dc0-90197aa7cd83 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:42:04.254952 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1281/2575] 10.222.0.50 () {58 vars in 1233 bytes} [Tue Sep 13 16:42:04 2022] DELETE /identity/v3/projects/f49c227e708f477fb8e16ff80067a085 => generated 0 bytes in 144 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 16:42:04.258237 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5f8fa0f-10eb-47af-8dc0-90197aa7cd83 admin admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:42:04.260863 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-d5f8fa0f-10eb-47af-8dc0-90197aa7cd83 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:42:04.293356 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8e5a372-867c-41f0-9ea9-fde31fd9b00b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:42:04.293997 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8e5a372-867c-41f0-9ea9-fde31fd9b00b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:42:04.294276 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8e5a372-867c-41f0-9ea9-fde31fd9b00b admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:42:04.294548 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8e5a372-867c-41f0-9ea9-fde31fd9b00b admin admin] PATH_INFO: `/v3/projects/c2310d76bc7c40d8ac8e1276ce60126f` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:42:04.295712 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8e5a372-867c-41f0-9ea9-fde31fd9b00b admin admin] RBAC: Authorizing `identity:delete_project(project_id=c2310d76bc7c40d8ac8e1276ce60126f)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:42:04.300547 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8e5a372-867c-41f0-9ea9-fde31fd9b00b admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:42:04.349324 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5f8fa0f-10eb-47af-8dc0-90197aa7cd83 admin admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:42:04.355225 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-f8e5a372-867c-41f0-9ea9-fde31fd9b00b admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'c2310d76bc7c40d8ac8e1276ce60126f', 'request_id': 'req-f8e5a372-867c-41f0-9ea9-fde31fd9b00b'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:42:04.362583 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-f8e5a372-867c-41f0-9ea9-fde31fd9b00b admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'c2310d76bc7c40d8ac8e1276ce60126f', 'request_id': 'req-f8e5a372-867c-41f0-9ea9-fde31fd9b00b'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:42:04.364423 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=94,keystone:DELETE=5,keystone:INSERT=1 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:42:04.364938 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:42:04.372339 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-f8e5a372-867c-41f0-9ea9-fde31fd9b00b admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'c2310d76bc7c40d8ac8e1276ce60126f', 'request_id': 'req-f8e5a372-867c-41f0-9ea9-fde31fd9b00b'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:42:04.384139 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-f8e5a372-867c-41f0-9ea9-fde31fd9b00b admin admin] The token cache is being invalidate because project c2310d76bc7c40d8ac8e1276ce60126f was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81437) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 13 16:42:04.384418 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-f8e5a372-867c-41f0-9ea9-fde31fd9b00b admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:42:04.385086 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-f8e5a372-867c-41f0-9ea9-fde31fd9b00b admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:42:04.386565 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1295/2576] 10.222.0.50 () {58 vars in 1233 bytes} [Tue Sep 13 16:42:04 2022] DELETE /identity/v3/projects/c2310d76bc7c40d8ac8e1276ce60126f => generated 0 bytes in 206 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 13 16:42:04.390326 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8e5a372-867c-41f0-9ea9-fde31fd9b00b admin admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:42:04.392457 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-f8e5a372-867c-41f0-9ea9-fde31fd9b00b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:42:04.424519 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffd77c3d-f346-4684-ac1a-67e11f737c73 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:42:04.425510 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffd77c3d-f346-4684-ac1a-67e11f737c73 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:42:04.425748 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffd77c3d-f346-4684-ac1a-67e11f737c73 admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:42:04.426021 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffd77c3d-f346-4684-ac1a-67e11f737c73 admin admin] PATH_INFO: `/v3/users/180031cda72f4862974b99219f71a9d5` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:42:04.427203 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffd77c3d-f346-4684-ac1a-67e11f737c73 admin admin] RBAC: Authorizing `identity:delete_user(user_id=180031cda72f4862974b99219f71a9d5)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:42:04.429112 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffd77c3d-f346-4684-ac1a-67e11f737c73 admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:42:04.465269 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8e5a372-867c-41f0-9ea9-fde31fd9b00b admin admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:42:04.474857 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49b5c2c1-b057-4214-8169-f5b5f8d59f3f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:42:04.475716 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49b5c2c1-b057-4214-8169-f5b5f8d59f3f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:42:04.476073 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49b5c2c1-b057-4214-8169-f5b5f8d59f3f admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:42:04.476547 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49b5c2c1-b057-4214-8169-f5b5f8d59f3f admin admin] PATH_INFO: `/v3/users/26066ccbf514493997feeb5892f823e6` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:42:04.477923 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49b5c2c1-b057-4214-8169-f5b5f8d59f3f admin admin] RBAC: Authorizing `identity:delete_user(user_id=26066ccbf514493997feeb5892f823e6)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:42:04.480051 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49b5c2c1-b057-4214-8169-f5b5f8d59f3f admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:42:04.483289 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-ffd77c3d-f346-4684-ac1a-67e11f737c73 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '180031cda72f4862974b99219f71a9d5'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:42:04.493453 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-ffd77c3d-f346-4684-ac1a-67e11f737c73 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '180031cda72f4862974b99219f71a9d5'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:42:04.509388 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-ffd77c3d-f346-4684-ac1a-67e11f737c73 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '180031cda72f4862974b99219f71a9d5'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:42:04.510359 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-ffd77c3d-f346-4684-ac1a-67e11f737c73 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '180031cda72f4862974b99219f71a9d5'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:42:04.523298 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-ffd77c3d-f346-4684-ac1a-67e11f737c73 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '180031cda72f4862974b99219f71a9d5'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:42:04.525894 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1282/2577] 10.222.0.50 () {58 vars in 1224 bytes} [Tue Sep 13 16:42:04 2022] DELETE /identity/v3/users/180031cda72f4862974b99219f71a9d5 => generated 0 bytes in 267 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 13 16:42:04.551037 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-49b5c2c1-b057-4214-8169-f5b5f8d59f3f admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '26066ccbf514493997feeb5892f823e6'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:42:04.562735 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-49b5c2c1-b057-4214-8169-f5b5f8d59f3f admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '26066ccbf514493997feeb5892f823e6'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:42:04.577844 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-49b5c2c1-b057-4214-8169-f5b5f8d59f3f admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '26066ccbf514493997feeb5892f823e6'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:42:04.578671 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-49b5c2c1-b057-4214-8169-f5b5f8d59f3f admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '26066ccbf514493997feeb5892f823e6'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:42:04.587201 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-49b5c2c1-b057-4214-8169-f5b5f8d59f3f admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '26066ccbf514493997feeb5892f823e6'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:42:04.594237 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1296/2578] 10.222.0.50 () {58 vars in 1224 bytes} [Tue Sep 13 16:42:04 2022] DELETE /identity/v3/users/26066ccbf514493997feeb5892f823e6 => generated 0 bytes in 204 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 13 16:42:04.842560 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffd77c3d-f346-4684-ac1a-67e11f737c73 admin admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:42:04.845117 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-ffd77c3d-f346-4684-ac1a-67e11f737c73 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:42:04.923721 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49b5c2c1-b057-4214-8169-f5b5f8d59f3f admin admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:42:04.925191 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-49b5c2c1-b057-4214-8169-f5b5f8d59f3f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:42:04.957966 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffd77c3d-f346-4684-ac1a-67e11f737c73 admin admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:42:05.010328 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:42:05.019587 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49b5c2c1-b057-4214-8169-f5b5f8d59f3f admin admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:42:05.025342 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb148b67-f71d-45b5-9d63-ed56501e70cb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:42:05.026108 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb148b67-f71d-45b5-9d63-ed56501e70cb admin admin] REQUEST_METHOD: `DELETE` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:42:05.026395 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb148b67-f71d-45b5-9d63-ed56501e70cb admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:42:05.026678 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb148b67-f71d-45b5-9d63-ed56501e70cb admin admin] PATH_INFO: `/v3/projects/ccabb33344594dfda157d06b1cb5342e` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:42:05.027819 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb148b67-f71d-45b5-9d63-ed56501e70cb admin admin] RBAC: Authorizing `identity:delete_project(project_id=ccabb33344594dfda157d06b1cb5342e)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:42:05.030019 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb148b67-f71d-45b5-9d63-ed56501e70cb admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:42:05.034015 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29fe0642-64dd-4aa8-821d-13701e3c8970 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:42:05.034677 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29fe0642-64dd-4aa8-821d-13701e3c8970 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:42:05.034970 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29fe0642-64dd-4aa8-821d-13701e3c8970 admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:42:05.035243 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29fe0642-64dd-4aa8-821d-13701e3c8970 admin admin] PATH_INFO: `/v3/projects/b8a301bb55ff4b5f81701e0d843614f9` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:42:05.036411 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29fe0642-64dd-4aa8-821d-13701e3c8970 admin admin] RBAC: Authorizing `identity:delete_project(project_id=b8a301bb55ff4b5f81701e0d843614f9)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:42:05.038025 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=87,keystone:DELETE=12,keystone:INSERT=1 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:42:05.038816 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29fe0642-64dd-4aa8-821d-13701e3c8970 admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:42:05.091588 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-bb148b67-f71d-45b5-9d63-ed56501e70cb admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'ccabb33344594dfda157d06b1cb5342e', 'request_id': 'req-bb148b67-f71d-45b5-9d63-ed56501e70cb'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:42:05.097130 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-29fe0642-64dd-4aa8-821d-13701e3c8970 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'b8a301bb55ff4b5f81701e0d843614f9', 'request_id': 'req-29fe0642-64dd-4aa8-821d-13701e3c8970'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:42:05.100455 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-bb148b67-f71d-45b5-9d63-ed56501e70cb admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'ccabb33344594dfda157d06b1cb5342e', 'request_id': 'req-bb148b67-f71d-45b5-9d63-ed56501e70cb'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:42:05.104563 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-29fe0642-64dd-4aa8-821d-13701e3c8970 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'b8a301bb55ff4b5f81701e0d843614f9', 'request_id': 'req-29fe0642-64dd-4aa8-821d-13701e3c8970'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:42:05.108750 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-bb148b67-f71d-45b5-9d63-ed56501e70cb admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'ccabb33344594dfda157d06b1cb5342e', 'request_id': 'req-bb148b67-f71d-45b5-9d63-ed56501e70cb'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:42:05.110496 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-29fe0642-64dd-4aa8-821d-13701e3c8970 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'b8a301bb55ff4b5f81701e0d843614f9', 'request_id': 'req-29fe0642-64dd-4aa8-821d-13701e3c8970'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:42:05.120907 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-bb148b67-f71d-45b5-9d63-ed56501e70cb admin admin] The token cache is being invalidate because project ccabb33344594dfda157d06b1cb5342e was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81437) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 13 16:42:05.121221 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-bb148b67-f71d-45b5-9d63-ed56501e70cb admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:42:05.122011 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-bb148b67-f71d-45b5-9d63-ed56501e70cb admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:42:05.122141 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-29fe0642-64dd-4aa8-821d-13701e3c8970 admin admin] The token cache is being invalidate because project b8a301bb55ff4b5f81701e0d843614f9 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81438) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 13 16:42:05.122303 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-29fe0642-64dd-4aa8-821d-13701e3c8970 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:42:05.122995 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-29fe0642-64dd-4aa8-821d-13701e3c8970 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:42:05.123886 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1297/2579] 10.222.0.50 () {58 vars in 1233 bytes} [Tue Sep 13 16:42:04 2022] DELETE /identity/v3/projects/ccabb33344594dfda157d06b1cb5342e => generated 0 bytes in 201 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 13 16:42:05.124513 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1283/2580] 10.222.0.50 () {58 vars in 1233 bytes} [Tue Sep 13 16:42:04 2022] DELETE /identity/v3/projects/b8a301bb55ff4b5f81701e0d843614f9 => generated 0 bytes in 283 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 16:42:05.128308 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb148b67-f71d-45b5-9d63-ed56501e70cb admin admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:42:05.130146 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87ce0fcf-7584-4f5d-88cc-b0d93f167148 None None] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:42:05.130267 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-bb148b67-f71d-45b5-9d63-ed56501e70cb admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:42:05.130430 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87ce0fcf-7584-4f5d-88cc-b0d93f167148 None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:42:05.130767 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87ce0fcf-7584-4f5d-88cc-b0d93f167148 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:42:05.151391 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.auth.core [None req-87ce0fcf-7584-4f5d-88cc-b0d93f167148 None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81438) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:42:05.152625 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-87ce0fcf-7584-4f5d-88cc-b0d93f167148 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:42:05.196366 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb148b67-f71d-45b5-9d63-ed56501e70cb admin admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:42:05.216682 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1284/2581] 10.222.0.50 () {58 vars in 958 bytes} [Tue Sep 13 16:42:05 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 13 16:42:05.225447 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7fb9cdd-4e46-4f68-8e40-471c897bf5af None None] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:42:05.225724 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7fb9cdd-4e46-4f68-8e40-471c897bf5af None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:42:05.225993 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7fb9cdd-4e46-4f68-8e40-471c897bf5af None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:42:05.249027 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.auth.core [None req-f7fb9cdd-4e46-4f68-8e40-471c897bf5af None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81438) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:42:05.250366 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-f7fb9cdd-4e46-4f68-8e40-471c897bf5af None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:42:05.259055 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1285/2582] 10.222.0.50 () {58 vars in 958 bytes} [Tue Sep 13 16:42:05 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 35 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 13 16:42:05.261536 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b81a92e7-dafd-4b8c-9b4d-7dfd783886e2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:42:05.262180 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b81a92e7-dafd-4b8c-9b4d-7dfd783886e2 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:42:05.262511 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b81a92e7-dafd-4b8c-9b4d-7dfd783886e2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:42:05.262742 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b81a92e7-dafd-4b8c-9b4d-7dfd783886e2 admin admin] PATH_INFO: `/v3/users/382addbce3f04db8a648ecaf2a23a4f2` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:42:05.263346 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7fb9cdd-4e46-4f68-8e40-471c897bf5af None None] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:42:05.263932 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b81a92e7-dafd-4b8c-9b4d-7dfd783886e2 admin admin] RBAC: Authorizing `identity:delete_user(user_id=382addbce3f04db8a648ecaf2a23a4f2)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:42:05.265905 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b81a92e7-dafd-4b8c-9b4d-7dfd783886e2 admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:42:05.272886 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7fb9cdd-4e46-4f68-8e40-471c897bf5af None None] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:42:05.277500 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4a8b477-79f5-4023-94c1-d37f1039a26e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:42:05.278086 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4a8b477-79f5-4023-94c1-d37f1039a26e admin admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:42:05.278366 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4a8b477-79f5-4023-94c1-d37f1039a26e admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:42:05.278642 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4a8b477-79f5-4023-94c1-d37f1039a26e admin admin] PATH_INFO: `/v3/domains` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:42:05.279313 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4a8b477-79f5-4023-94c1-d37f1039a26e admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81438) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 16:42:05.279619 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4a8b477-79f5-4023-94c1-d37f1039a26e admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:42:05.281224 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4a8b477-79f5-4023-94c1-d37f1039a26e admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:42:05.293711 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1286/2583] 10.222.0.50 () {58 vars in 1153 bytes} [Tue Sep 13 16:42:05 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 13 16:42:05.298488 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4a8b477-79f5-4023-94c1-d37f1039a26e admin admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:42:05.308708 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4a8b477-79f5-4023-94c1-d37f1039a26e admin admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:42:05.313422 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b2507bb-2ea9-41ec-a2d0-8ceff40ba9cf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:42:05.314018 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b2507bb-2ea9-41ec-a2d0-8ceff40ba9cf admin admin] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:42:05.314322 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b2507bb-2ea9-41ec-a2d0-8ceff40ba9cf admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:42:05.314601 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b2507bb-2ea9-41ec-a2d0-8ceff40ba9cf admin admin] PATH_INFO: `/v3/projects` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:42:05.315247 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b2507bb-2ea9-41ec-a2d0-8ceff40ba9cf admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:42:05.316344 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=86,keystone:DELETE=13,keystone:INSERT=1 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:42:05.317141 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b2507bb-2ea9-41ec-a2d0-8ceff40ba9cf admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:42:05.318865 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-b81a92e7-dafd-4b8c-9b4d-7dfd783886e2 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '382addbce3f04db8a648ecaf2a23a4f2'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:42:05.327020 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-b81a92e7-dafd-4b8c-9b4d-7dfd783886e2 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '382addbce3f04db8a648ecaf2a23a4f2'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:42:05.340789 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-b81a92e7-dafd-4b8c-9b4d-7dfd783886e2 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '382addbce3f04db8a648ecaf2a23a4f2'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:42:05.341470 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-b81a92e7-dafd-4b8c-9b4d-7dfd783886e2 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '382addbce3f04db8a648ecaf2a23a4f2'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:42:05.346096 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1287/2584] 10.222.0.50 () {60 vars in 1153 bytes} [Tue Sep 13 16:42:05 2022] POST /identity/v3/projects => generated 381 bytes in 48 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 13 16:42:05.350186 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-b81a92e7-dafd-4b8c-9b4d-7dfd783886e2 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '382addbce3f04db8a648ecaf2a23a4f2'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:42:05.350332 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b2507bb-2ea9-41ec-a2d0-8ceff40ba9cf admin admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:42:05.351852 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-4b2507bb-2ea9-41ec-a2d0-8ceff40ba9cf admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:42:05.356344 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1298/2585] 10.222.0.50 () {58 vars in 1224 bytes} [Tue Sep 13 16:42:05 2022] DELETE /identity/v3/users/382addbce3f04db8a648ecaf2a23a4f2 => generated 0 bytes in 229 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 16:42:05.405164 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:42:05.452318 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b2507bb-2ea9-41ec-a2d0-8ceff40ba9cf admin admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:42:05.511882 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-304caa6c-052f-4614-8732-0a0038b3d2dd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:42:05.512489 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-304caa6c-052f-4614-8732-0a0038b3d2dd admin admin] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:42:05.512832 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-304caa6c-052f-4614-8732-0a0038b3d2dd admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:42:05.513124 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-304caa6c-052f-4614-8732-0a0038b3d2dd admin admin] PATH_INFO: `/v3/users` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:42:05.513728 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-304caa6c-052f-4614-8732-0a0038b3d2dd admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:42:05.515739 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-304caa6c-052f-4614-8732-0a0038b3d2dd admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:42:05.548548 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1288/2586] 10.222.0.50 () {60 vars in 1144 bytes} [Tue Sep 13 16:42:05 2022] POST /identity/v3/users => generated 291 bytes in 199 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 13 16:42:05.552500 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b81a92e7-dafd-4b8c-9b4d-7dfd783886e2 admin admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:42:05.564815 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b81a92e7-dafd-4b8c-9b4d-7dfd783886e2 admin admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:42:05.569912 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c4e0010-8ae9-4753-94a0-00cf8f2429b8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:42:05.570530 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c4e0010-8ae9-4753-94a0-00cf8f2429b8 admin admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:42:05.570814 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c4e0010-8ae9-4753-94a0-00cf8f2429b8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:42:05.571516 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c4e0010-8ae9-4753-94a0-00cf8f2429b8 admin admin] PATH_INFO: `/v3/roles` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:42:05.571778 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c4e0010-8ae9-4753-94a0-00cf8f2429b8 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:42:05.573400 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c4e0010-8ae9-4753-94a0-00cf8f2429b8 admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:42:05.585276 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1299/2587] 10.222.0.50 () {58 vars in 1122 bytes} [Tue Sep 13 16:42:05 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 16:42:05.588955 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-304caa6c-052f-4614-8732-0a0038b3d2dd admin admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:42:05.599995 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-304caa6c-052f-4614-8732-0a0038b3d2dd admin admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:42:05.604921 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4a3f514-e5ee-4c3c-91ad-88aae40db2f2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:42:05.605523 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4a3f514-e5ee-4c3c-91ad-88aae40db2f2 admin admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:42:05.605842 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4a3f514-e5ee-4c3c-91ad-88aae40db2f2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:42:05.606119 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4a3f514-e5ee-4c3c-91ad-88aae40db2f2 admin admin] PATH_INFO: `/v3/roles` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:42:05.606944 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4a3f514-e5ee-4c3c-91ad-88aae40db2f2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:42:05.608574 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4a3f514-e5ee-4c3c-91ad-88aae40db2f2 admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:42:05.619559 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1289/2588] 10.222.0.50 () {58 vars in 1122 bytes} [Tue Sep 13 16:42:05 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 16:42:05.624302 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c4e0010-8ae9-4753-94a0-00cf8f2429b8 admin admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:42:05.636087 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c4e0010-8ae9-4753-94a0-00cf8f2429b8 admin admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:42:05.640551 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6d1c303-326e-4a59-99a7-b89bfd5bec24 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:42:05.641201 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6d1c303-326e-4a59-99a7-b89bfd5bec24 admin admin] REQUEST_METHOD: `PUT` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:42:05.642269 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6d1c303-326e-4a59-99a7-b89bfd5bec24 admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:42:05.642553 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6d1c303-326e-4a59-99a7-b89bfd5bec24 admin admin] PATH_INFO: `/v3/projects/54b78d5cc7e9452f9694afe2bb869174/users/0e2becf5a7704a7f9aca6e1ae6f7f340/roles/15db4aaca3264ae29bde38903ba43471` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:42:05.661645 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6d1c303-326e-4a59-99a7-b89bfd5bec24 admin admin] RBAC: Authorizing `identity:create_grant(project_id=54b78d5cc7e9452f9694afe2bb869174, user_id=0e2becf5a7704a7f9aca6e1ae6f7f340, role_id=15db4aaca3264ae29bde38903ba43471)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:42:05.664797 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6d1c303-326e-4a59-99a7-b89bfd5bec24 admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:42:05.676178 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4a3f514-e5ee-4c3c-91ad-88aae40db2f2 admin admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:42:05.678219 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-d4a3f514-e5ee-4c3c-91ad-88aae40db2f2 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:42:05.681425 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1300/2589] 10.222.0.50 () {60 vars in 1483 bytes} [Tue Sep 13 16:42:05 2022] PUT /identity/v3/projects/54b78d5cc7e9452f9694afe2bb869174/users/0e2becf5a7704a7f9aca6e1ae6f7f340/roles/15db4aaca3264ae29bde38903ba43471 => generated 0 bytes in 58 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 16:42:05.697204 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e21b104c-ee01-4832-b781-e7153947cc43 None None] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:42:05.697415 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e21b104c-ee01-4832-b781-e7153947cc43 None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:42:05.697694 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e21b104c-ee01-4832-b781-e7153947cc43 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:42:05.706613 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4a3f514-e5ee-4c3c-91ad-88aae40db2f2 admin admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:42:05.728770 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.auth.core [None req-e21b104c-ee01-4832-b781-e7153947cc43 None None] MFA Rules not processed for user `0e2becf5a7704a7f9aca6e1ae6f7f340`. Rule list: `[]` (Enabled: `True`). {{(pid=81437) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:42:05.729996 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-e21b104c-ee01-4832-b781-e7153947cc43 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:42:05.771277 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc289c17-90e5-42f0-92bc-d65aa70a4913 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:42:05.772023 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc289c17-90e5-42f0-92bc-d65aa70a4913 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:42:05.772307 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc289c17-90e5-42f0-92bc-d65aa70a4913 admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:42:05.772580 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc289c17-90e5-42f0-92bc-d65aa70a4913 admin admin] PATH_INFO: `/v3/projects/f2dd36092ffe45eb9fe245ec2af9d3a9` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:42:05.773769 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc289c17-90e5-42f0-92bc-d65aa70a4913 admin admin] RBAC: Authorizing `identity:delete_project(project_id=f2dd36092ffe45eb9fe245ec2af9d3a9)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:42:05.775736 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc289c17-90e5-42f0-92bc-d65aa70a4913 admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:42:05.801747 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1301/2590] 10.222.0.50 () {58 vars in 958 bytes} [Tue Sep 13 16:42:05 2022] POST /identity/v3/auth/tokens => generated 3258 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 13 16:42:05.806179 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e21b104c-ee01-4832-b781-e7153947cc43 None None] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:42:05.817565 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e21b104c-ee01-4832-b781-e7153947cc43 None None] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:42:05.829202 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-cc289c17-90e5-42f0-92bc-d65aa70a4913 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'f2dd36092ffe45eb9fe245ec2af9d3a9', 'request_id': 'req-cc289c17-90e5-42f0-92bc-d65aa70a4913'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:42:05.836728 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-cc289c17-90e5-42f0-92bc-d65aa70a4913 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'f2dd36092ffe45eb9fe245ec2af9d3a9', 'request_id': 'req-cc289c17-90e5-42f0-92bc-d65aa70a4913'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:42:05.839572 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=90,keystone:INSERT=4,keystone:DELETE=6 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:42:05.844684 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-cc289c17-90e5-42f0-92bc-d65aa70a4913 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'f2dd36092ffe45eb9fe245ec2af9d3a9', 'request_id': 'req-cc289c17-90e5-42f0-92bc-d65aa70a4913'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:42:05.859278 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-cc289c17-90e5-42f0-92bc-d65aa70a4913 admin admin] The token cache is being invalidate because project f2dd36092ffe45eb9fe245ec2af9d3a9 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81438) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 13 16:42:05.859598 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-cc289c17-90e5-42f0-92bc-d65aa70a4913 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:42:05.861407 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-cc289c17-90e5-42f0-92bc-d65aa70a4913 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:42:05.863590 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1290/2591] 10.222.0.50 () {58 vars in 1233 bytes} [Tue Sep 13 16:42:05 2022] DELETE /identity/v3/projects/f2dd36092ffe45eb9fe245ec2af9d3a9 => generated 0 bytes in 188 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 16:42:05.869388 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c9ea475-e37a-47c0-aa74-38468f7a4c80 None None] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:42:05.869673 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c9ea475-e37a-47c0-aa74-38468f7a4c80 None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:42:05.869948 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c9ea475-e37a-47c0-aa74-38468f7a4c80 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:42:05.885581 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32f0e36e-91b6-42f9-9859-d76724f8f1b5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:42:05.886419 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32f0e36e-91b6-42f9-9859-d76724f8f1b5 admin admin] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:42:05.886711 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32f0e36e-91b6-42f9-9859-d76724f8f1b5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:42:05.887158 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32f0e36e-91b6-42f9-9859-d76724f8f1b5 admin admin] PATH_INFO: `/v3/projects` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:42:05.888309 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32f0e36e-91b6-42f9-9859-d76724f8f1b5 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:42:05.890721 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32f0e36e-91b6-42f9-9859-d76724f8f1b5 admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:42:05.893777 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.auth.core [None req-4c9ea475-e37a-47c0-aa74-38468f7a4c80 None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81438) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:42:05.895105 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-4c9ea475-e37a-47c0-aa74-38468f7a4c80 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:42:05.911063 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1291/2592] 10.222.0.50 () {58 vars in 958 bytes} [Tue Sep 13 16:42:05 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 42 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:42:05.918437 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:DELETE=4,keystone:SELECT=93,keystone:INSERT=3 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:42:05.923321 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d38d849-dec8-40a2-8f19-c0b453bc7411 None None] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:42:05.923504 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d38d849-dec8-40a2-8f19-c0b453bc7411 None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:42:05.924021 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d38d849-dec8-40a2-8f19-c0b453bc7411 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:42:05.931850 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1302/2593] 10.222.0.50 () {60 vars in 1153 bytes} [Tue Sep 13 16:42:05 2022] POST /identity/v3/projects => generated 383 bytes in 125 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 13 16:42:05.932574 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32f0e36e-91b6-42f9-9859-d76724f8f1b5 admin admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:42:05.934851 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-32f0e36e-91b6-42f9-9859-d76724f8f1b5 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:42:05.946452 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.auth.core [None req-5d38d849-dec8-40a2-8f19-c0b453bc7411 None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81438) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:42:05.948314 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-5d38d849-dec8-40a2-8f19-c0b453bc7411 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:42:06.020983 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1292/2594] 10.222.0.50 () {58 vars in 958 bytes} [Tue Sep 13 16:42:05 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 13 16:42:06.025569 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d38d849-dec8-40a2-8f19-c0b453bc7411 None None] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:42:06.028932 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32f0e36e-91b6-42f9-9859-d76724f8f1b5 admin admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:42:06.035775 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50438224-8716-4d3e-8b9b-6d8a6f87596f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:42:06.036429 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d38d849-dec8-40a2-8f19-c0b453bc7411 None None] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:42:06.036550 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50438224-8716-4d3e-8b9b-6d8a6f87596f admin admin] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:42:06.036946 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50438224-8716-4d3e-8b9b-6d8a6f87596f admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:42:06.037327 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50438224-8716-4d3e-8b9b-6d8a6f87596f admin admin] PATH_INFO: `/v3/users` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:42:06.038081 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50438224-8716-4d3e-8b9b-6d8a6f87596f admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:42:06.040858 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50438224-8716-4d3e-8b9b-6d8a6f87596f admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:42:06.042654 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb6dfb43-c69e-43a8-9370-8df341ce2017 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:42:06.043774 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb6dfb43-c69e-43a8-9370-8df341ce2017 admin admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:42:06.044289 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb6dfb43-c69e-43a8-9370-8df341ce2017 admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:42:06.044696 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb6dfb43-c69e-43a8-9370-8df341ce2017 admin admin] PATH_INFO: `/v3/domains` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:42:06.046055 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb6dfb43-c69e-43a8-9370-8df341ce2017 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81438) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 16:42:06.046644 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb6dfb43-c69e-43a8-9370-8df341ce2017 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:42:06.049631 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb6dfb43-c69e-43a8-9370-8df341ce2017 admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:42:06.071892 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1293/2595] 10.222.0.50 () {58 vars in 1153 bytes} [Tue Sep 13 16:42:06 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 47 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 16:42:06.076714 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1303/2596] 10.222.0.50 () {60 vars in 1144 bytes} [Tue Sep 13 16:42:05 2022] POST /identity/v3/users => generated 298 bytes in 144 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 16:42:06.076948 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb6dfb43-c69e-43a8-9370-8df341ce2017 admin admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:42:06.077856 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50438224-8716-4d3e-8b9b-6d8a6f87596f admin admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:42:06.088442 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb6dfb43-c69e-43a8-9370-8df341ce2017 admin admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:42:06.094100 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50438224-8716-4d3e-8b9b-6d8a6f87596f admin admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:42:06.094349 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d862d6e4-0823-4d00-b789-83bc301883bd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:42:06.095178 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d862d6e4-0823-4d00-b789-83bc301883bd admin admin] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:42:06.095460 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d862d6e4-0823-4d00-b789-83bc301883bd admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:42:06.095732 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d862d6e4-0823-4d00-b789-83bc301883bd admin admin] PATH_INFO: `/v3/projects` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:42:06.096370 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d862d6e4-0823-4d00-b789-83bc301883bd admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:42:06.098268 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d862d6e4-0823-4d00-b789-83bc301883bd admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:42:06.099717 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05a47b53-d526-4ac3-899c-cca149774f8d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:42:06.100399 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05a47b53-d526-4ac3-899c-cca149774f8d admin admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:42:06.100844 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05a47b53-d526-4ac3-899c-cca149774f8d admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:42:06.101041 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05a47b53-d526-4ac3-899c-cca149774f8d admin admin] PATH_INFO: `/v3/roles` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:42:06.102089 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05a47b53-d526-4ac3-899c-cca149774f8d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:42:06.105449 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05a47b53-d526-4ac3-899c-cca149774f8d admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:42:06.118759 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1304/2597] 10.222.0.50 () {58 vars in 1122 bytes} [Tue Sep 13 16:42:06 2022] GET /identity/v3/roles => generated 1420 bytes in 41 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 16:42:06.123547 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05a47b53-d526-4ac3-899c-cca149774f8d admin admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:42:06.132167 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1294/2598] 10.222.0.50 () {60 vars in 1153 bytes} [Tue Sep 13 16:42:06 2022] POST /identity/v3/projects => generated 385 bytes in 56 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 16:42:06.136233 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d862d6e4-0823-4d00-b789-83bc301883bd admin admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:42:06.136374 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05a47b53-d526-4ac3-899c-cca149774f8d admin admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:42:06.209298 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b220e61-227b-47c6-90ab-37d6a2106243 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:42:06.209298 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b220e61-227b-47c6-90ab-37d6a2106243 admin admin] REQUEST_METHOD: `PUT` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:42:06.209969 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b220e61-227b-47c6-90ab-37d6a2106243 admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:42:06.209969 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b220e61-227b-47c6-90ab-37d6a2106243 admin admin] PATH_INFO: `/v3/projects/e400cf6cc595413cb1b18435e94fd903/users/16e0c1dffbe54278863f506e11d59f36/roles/031d4e1a23e34f71858fc4a6dfbd4ccb` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:42:06.216953 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d862d6e4-0823-4d00-b789-83bc301883bd admin admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:42:06.222211 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2eb528a1-f075-48d1-b1f3-27af4324c150 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:42:06.222816 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2eb528a1-f075-48d1-b1f3-27af4324c150 admin admin] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:42:06.223146 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2eb528a1-f075-48d1-b1f3-27af4324c150 admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:42:06.223417 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2eb528a1-f075-48d1-b1f3-27af4324c150 admin admin] PATH_INFO: `/v3/users` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:42:06.224101 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2eb528a1-f075-48d1-b1f3-27af4324c150 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:42:06.225550 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b220e61-227b-47c6-90ab-37d6a2106243 admin admin] RBAC: Authorizing `identity:create_grant(project_id=e400cf6cc595413cb1b18435e94fd903, user_id=16e0c1dffbe54278863f506e11d59f36, role_id=031d4e1a23e34f71858fc4a6dfbd4ccb)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:42:06.226029 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2eb528a1-f075-48d1-b1f3-27af4324c150 admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:42:06.228273 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b220e61-227b-47c6-90ab-37d6a2106243 admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:42:06.244086 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1305/2599] 10.222.0.50 () {60 vars in 1483 bytes} [Tue Sep 13 16:42:06 2022] PUT /identity/v3/projects/e400cf6cc595413cb1b18435e94fd903/users/16e0c1dffbe54278863f506e11d59f36/roles/031d4e1a23e34f71858fc4a6dfbd4ccb => generated 0 bytes in 121 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 16:42:06.245068 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b220e61-227b-47c6-90ab-37d6a2106243 admin admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:42:06.246705 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-5b220e61-227b-47c6-90ab-37d6a2106243 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:42:06.252684 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1295/2600] 10.222.0.50 () {60 vars in 1144 bytes} [Tue Sep 13 16:42:06 2022] POST /identity/v3/users => generated 293 bytes in 117 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 13 16:42:06.253391 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2eb528a1-f075-48d1-b1f3-27af4324c150 admin admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:42:06.265439 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2eb528a1-f075-48d1-b1f3-27af4324c150 admin admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:42:06.301807 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=95,keystone:DELETE=1,keystone:INSERT=4 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:42:06.312696 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b220e61-227b-47c6-90ab-37d6a2106243 admin admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:42:06.318498 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-520e4b78-f517-41b3-a38f-79cb837339e1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:42:06.319155 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-520e4b78-f517-41b3-a38f-79cb837339e1 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:42:06.319436 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-520e4b78-f517-41b3-a38f-79cb837339e1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:42:06.319707 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-520e4b78-f517-41b3-a38f-79cb837339e1 admin admin] PATH_INFO: `/v3/users/ba7c69b25caa4529994e48856cb3a3c6` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:42:06.321269 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-520e4b78-f517-41b3-a38f-79cb837339e1 admin admin] RBAC: Authorizing `identity:delete_user(user_id=ba7c69b25caa4529994e48856cb3a3c6)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:42:06.323225 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-520e4b78-f517-41b3-a38f-79cb837339e1 admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:42:06.325557 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:42:06.339787 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1b0f13b-62f7-486b-b690-d20806094cbb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:42:06.340426 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1b0f13b-62f7-486b-b690-d20806094cbb admin admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:42:06.340831 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1b0f13b-62f7-486b-b690-d20806094cbb admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:42:06.341113 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1b0f13b-62f7-486b-b690-d20806094cbb admin admin] PATH_INFO: `/v3/roles` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:42:06.342384 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1b0f13b-62f7-486b-b690-d20806094cbb admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:42:06.344984 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1b0f13b-62f7-486b-b690-d20806094cbb admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:42:06.360735 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1296/2601] 10.222.0.50 () {58 vars in 1122 bytes} [Tue Sep 13 16:42:06 2022] GET /identity/v3/roles => generated 1420 bytes in 105 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 16:42:06.360735 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1b0f13b-62f7-486b-b690-d20806094cbb admin admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:42:06.369216 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1b0f13b-62f7-486b-b690-d20806094cbb admin admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:42:06.374574 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-311d8e3e-9fd1-4402-a83d-4a79c16b4f47 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:42:06.375183 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-311d8e3e-9fd1-4402-a83d-4a79c16b4f47 admin admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:42:06.375478 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-311d8e3e-9fd1-4402-a83d-4a79c16b4f47 admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:42:06.376326 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-311d8e3e-9fd1-4402-a83d-4a79c16b4f47 admin admin] PATH_INFO: `/v3/roles` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:42:06.376962 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-311d8e3e-9fd1-4402-a83d-4a79c16b4f47 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:42:06.377314 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-520e4b78-f517-41b3-a38f-79cb837339e1 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'ba7c69b25caa4529994e48856cb3a3c6'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:42:06.378562 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-311d8e3e-9fd1-4402-a83d-4a79c16b4f47 admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:42:06.384972 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-520e4b78-f517-41b3-a38f-79cb837339e1 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'ba7c69b25caa4529994e48856cb3a3c6'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:42:06.388362 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1297/2602] 10.222.0.50 () {58 vars in 1122 bytes} [Tue Sep 13 16:42:06 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 16:42:06.389214 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-311d8e3e-9fd1-4402-a83d-4a79c16b4f47 admin admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:42:06.399308 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-311d8e3e-9fd1-4402-a83d-4a79c16b4f47 admin admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:42:06.400755 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-520e4b78-f517-41b3-a38f-79cb837339e1 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'ba7c69b25caa4529994e48856cb3a3c6'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:42:06.401441 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-520e4b78-f517-41b3-a38f-79cb837339e1 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'ba7c69b25caa4529994e48856cb3a3c6'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:42:06.403994 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35f50dd3-5b7f-499c-9381-4a6cba772233 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:42:06.404701 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35f50dd3-5b7f-499c-9381-4a6cba772233 admin admin] REQUEST_METHOD: `PUT` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:42:06.405055 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35f50dd3-5b7f-499c-9381-4a6cba772233 admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:42:06.405402 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35f50dd3-5b7f-499c-9381-4a6cba772233 admin admin] PATH_INFO: `/v3/projects/e400cf6cc595413cb1b18435e94fd903/users/16e0c1dffbe54278863f506e11d59f36/roles/031d4e1a23e34f71858fc4a6dfbd4ccb` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:42:06.407480 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35f50dd3-5b7f-499c-9381-4a6cba772233 admin admin] RBAC: Authorizing `identity:create_grant(project_id=e400cf6cc595413cb1b18435e94fd903, user_id=16e0c1dffbe54278863f506e11d59f36, role_id=031d4e1a23e34f71858fc4a6dfbd4ccb)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:42:06.408447 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-520e4b78-f517-41b3-a38f-79cb837339e1 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'ba7c69b25caa4529994e48856cb3a3c6'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:42:06.410217 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35f50dd3-5b7f-499c-9381-4a6cba772233 admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:42:06.415972 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1306/2603] 10.222.0.50 () {58 vars in 1224 bytes} [Tue Sep 13 16:42:06 2022] DELETE /identity/v3/users/ba7c69b25caa4529994e48856cb3a3c6 => generated 0 bytes in 171 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 16:42:06.416713 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-520e4b78-f517-41b3-a38f-79cb837339e1 admin admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:42:06.418086 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-520e4b78-f517-41b3-a38f-79cb837339e1 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:42:06.424325 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1298/2604] 10.222.0.50 () {60 vars in 1483 bytes} [Tue Sep 13 16:42:06 2022] PUT /identity/v3/projects/e400cf6cc595413cb1b18435e94fd903/users/16e0c1dffbe54278863f506e11d59f36/roles/031d4e1a23e34f71858fc4a6dfbd4ccb => generated 0 bytes in 35 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 16:42:06.431719 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f92a8a63-03f1-4b2c-8eda-5f978273d665 None None] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:42:06.431719 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f92a8a63-03f1-4b2c-8eda-5f978273d665 None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:42:06.431719 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f92a8a63-03f1-4b2c-8eda-5f978273d665 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:42:06.447342 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.auth.core [None req-f92a8a63-03f1-4b2c-8eda-5f978273d665 None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81438) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:42:06.448980 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-f92a8a63-03f1-4b2c-8eda-5f978273d665 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:42:06.512739 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-520e4b78-f517-41b3-a38f-79cb837339e1 admin admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:42:06.519595 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1299/2605] 10.222.0.50 () {58 vars in 958 bytes} [Tue Sep 13 16:42:06 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:42:06.520978 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-786dfff0-52c0-4988-8355-95e984cb17bb None None] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:42:06.521298 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-786dfff0-52c0-4988-8355-95e984cb17bb None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:42:06.521567 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-786dfff0-52c0-4988-8355-95e984cb17bb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:42:06.550547 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.auth.core [None req-786dfff0-52c0-4988-8355-95e984cb17bb None None] MFA Rules not processed for user `16e0c1dffbe54278863f506e11d59f36`. Rule list: `[]` (Enabled: `True`). {{(pid=81438) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:42:06.551867 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-786dfff0-52c0-4988-8355-95e984cb17bb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:42:06.575806 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cff427f-c3b3-4cbb-ace8-58f2f3c7089f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:42:06.576530 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cff427f-c3b3-4cbb-ace8-58f2f3c7089f admin admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:42:06.576814 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cff427f-c3b3-4cbb-ace8-58f2f3c7089f admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:42:06.577088 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cff427f-c3b3-4cbb-ace8-58f2f3c7089f admin admin] PATH_INFO: `/v3/roles` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:42:06.577728 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7cff427f-c3b3-4cbb-ace8-58f2f3c7089f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:42:06.579515 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7cff427f-c3b3-4cbb-ace8-58f2f3c7089f admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:42:06.595116 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1307/2606] 10.222.0.50 () {58 vars in 1122 bytes} [Tue Sep 13 16:42:06 2022] GET /identity/v3/roles => generated 1420 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 16:42:06.596748 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cff427f-c3b3-4cbb-ace8-58f2f3c7089f admin admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:42:06.602585 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-7cff427f-c3b3-4cbb-ace8-58f2f3c7089f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:42:06.630935 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1300/2607] 10.222.0.50 () {58 vars in 958 bytes} [Tue Sep 13 16:42:06 2022] POST /identity/v3/auth/tokens => generated 3327 bytes in 111 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:42:06.631829 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-786dfff0-52c0-4988-8355-95e984cb17bb None None] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:42:06.635439 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:42:06.643325 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-786dfff0-52c0-4988-8355-95e984cb17bb None None] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:42:06.649295 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a320628b-1bdc-4c9d-a84e-02ea7ea74cbb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:42:06.650039 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a320628b-1bdc-4c9d-a84e-02ea7ea74cbb admin admin] REQUEST_METHOD: `PUT` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:42:06.650389 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a320628b-1bdc-4c9d-a84e-02ea7ea74cbb admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:42:06.650727 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a320628b-1bdc-4c9d-a84e-02ea7ea74cbb admin admin] PATH_INFO: `/v3/projects/18456647e1c04144864ccd3ee60ea89a/users/c3c5559749a54790a5bf891c827e0ced/roles/15db4aaca3264ae29bde38903ba43471` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:42:06.666622 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a320628b-1bdc-4c9d-a84e-02ea7ea74cbb admin admin] RBAC: Authorizing `identity:create_grant(project_id=18456647e1c04144864ccd3ee60ea89a, user_id=c3c5559749a54790a5bf891c827e0ced, role_id=15db4aaca3264ae29bde38903ba43471)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:42:06.669321 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a320628b-1bdc-4c9d-a84e-02ea7ea74cbb admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:42:06.672140 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cff427f-c3b3-4cbb-ace8-58f2f3c7089f admin admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:42:06.676890 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76606d71-17db-41d9-993c-d1212e544bcb service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f436f07939d4855a21b195bc7566244', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:42:06.677499 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76606d71-17db-41d9-993c-d1212e544bcb service neutron] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:42:06.677796 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76606d71-17db-41d9-993c-d1212e544bcb service neutron] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:42:06.678051 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76606d71-17db-41d9-993c-d1212e544bcb service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:42:06.682999 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1301/2608] 10.222.0.50 () {60 vars in 1483 bytes} [Tue Sep 13 16:42:06 2022] PUT /identity/v3/projects/18456647e1c04144864ccd3ee60ea89a/users/c3c5559749a54790a5bf891c827e0ced/roles/15db4aaca3264ae29bde38903ba43471 => generated 0 bytes in 52 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 16:42:06.683994 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a320628b-1bdc-4c9d-a84e-02ea7ea74cbb admin admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:42:06.685545 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-a320628b-1bdc-4c9d-a84e-02ea7ea74cbb admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:42:06.692409 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76606d71-17db-41d9-993c-d1212e544bcb service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:42:06.694105 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76606d71-17db-41d9-993c-d1212e544bcb service neutron] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:42:06.697336 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:42:06.704847 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1308/2609] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 16:42:06 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 109 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 16:42:06.706561 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc680538-89ae-4cc4-865c-80b0f24a81ed None None] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:42:06.706730 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc680538-89ae-4cc4-865c-80b0f24a81ed None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:42:06.707122 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc680538-89ae-4cc4-865c-80b0f24a81ed None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:42:06.747314 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.auth.core [None req-dc680538-89ae-4cc4-865c-80b0f24a81ed None None] MFA Rules not processed for user `c3c5559749a54790a5bf891c827e0ced`. Rule list: `[]` (Enabled: `True`). {{(pid=81437) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:42:06.749169 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-dc680538-89ae-4cc4-865c-80b0f24a81ed None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:42:06.758131 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a320628b-1bdc-4c9d-a84e-02ea7ea74cbb admin admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:42:06.762184 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8be7b831-f60f-41e3-9474-8b077086fd91 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7fd6b5afd67b48f9b0f0597ccd4cf4df', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:42:06.762828 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8be7b831-f60f-41e3-9474-8b077086fd91 service cinder] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:42:06.763385 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8be7b831-f60f-41e3-9474-8b077086fd91 service cinder] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:42:06.763514 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8be7b831-f60f-41e3-9474-8b077086fd91 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:42:06.832366 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1309/2610] 10.222.0.50 () {58 vars in 958 bytes} [Tue Sep 13 16:42:06 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 127 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:42:06.835962 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8be7b831-f60f-41e3-9474-8b077086fd91 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:42:06.837182 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8be7b831-f60f-41e3-9474-8b077086fd91 service cinder] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:42:06.837412 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc680538-89ae-4cc4-865c-80b0f24a81ed None None] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:42:06.855641 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1302/2611] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 16:42:06 2022] GET /identity/v3/auth/tokens => generated 3327 bytes in 172 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 16:42:06.921044 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc680538-89ae-4cc4-865c-80b0f24a81ed None None] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:42:06.926946 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d86e48b-3729-42f0-87cd-e6b50d7743a1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:42:06.927560 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d86e48b-3729-42f0-87cd-e6b50d7743a1 admin admin] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:42:06.927843 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d86e48b-3729-42f0-87cd-e6b50d7743a1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:42:06.928116 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d86e48b-3729-42f0-87cd-e6b50d7743a1 admin admin] PATH_INFO: `/v3/projects` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:42:06.928742 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d86e48b-3729-42f0-87cd-e6b50d7743a1 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:42:06.930687 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d86e48b-3729-42f0-87cd-e6b50d7743a1 admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:42:06.967093 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1310/2612] 10.222.0.50 () {60 vars in 1153 bytes} [Tue Sep 13 16:42:06 2022] POST /identity/v3/projects => generated 381 bytes in 130 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 13 16:42:06.970863 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8be7b831-f60f-41e3-9474-8b077086fd91 service cinder] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:42:06.979275 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d86e48b-3729-42f0-87cd-e6b50d7743a1 admin admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:42:06.985339 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8be7b831-f60f-41e3-9474-8b077086fd91 service cinder] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:42:07.051596 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:42:07.068300 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d86e48b-3729-42f0-87cd-e6b50d7743a1 admin admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:42:07.072987 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-022b411c-2fc4-4b7c-9949-6364411c3447 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7fd6b5afd67b48f9b0f0597ccd4cf4df', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:42:07.073671 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-022b411c-2fc4-4b7c-9949-6364411c3447 service cinder] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:42:07.074025 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-022b411c-2fc4-4b7c-9949-6364411c3447 service cinder] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:42:07.074484 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-022b411c-2fc4-4b7c-9949-6364411c3447 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:42:07.076142 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb2bb208-542c-4110-990a-3482ed5bb0ba admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:42:07.076355 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-022b411c-2fc4-4b7c-9949-6364411c3447 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:42:07.076751 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb2bb208-542c-4110-990a-3482ed5bb0ba admin admin] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:42:07.077026 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb2bb208-542c-4110-990a-3482ed5bb0ba admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:42:07.077295 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb2bb208-542c-4110-990a-3482ed5bb0ba admin admin] PATH_INFO: `/v3/users` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:42:07.077975 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb2bb208-542c-4110-990a-3482ed5bb0ba admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:42:07.080740 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb2bb208-542c-4110-990a-3482ed5bb0ba admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:42:07.104088 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1303/2613] 10.222.0.50 () {60 vars in 1144 bytes} [Tue Sep 13 16:42:06 2022] POST /identity/v3/users => generated 291 bytes in 134 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 13 16:42:07.104912 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb2bb208-542c-4110-990a-3482ed5bb0ba admin admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:42:07.106545 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-cb2bb208-542c-4110-990a-3482ed5bb0ba admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:42:07.119154 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb2bb208-542c-4110-990a-3482ed5bb0ba admin admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:42:07.124841 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf1d40cb-54d3-4c6b-a3d1-27140ece3383 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:42:07.125614 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf1d40cb-54d3-4c6b-a3d1-27140ece3383 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:42:07.126669 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf1d40cb-54d3-4c6b-a3d1-27140ece3383 admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:42:07.127019 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf1d40cb-54d3-4c6b-a3d1-27140ece3383 admin admin] PATH_INFO: `/v3/projects/e4e1f8566af24fcaab73006738c35cce` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:42:07.128930 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf1d40cb-54d3-4c6b-a3d1-27140ece3383 admin admin] RBAC: Authorizing `identity:delete_project(project_id=e4e1f8566af24fcaab73006738c35cce)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:42:07.130335 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf1d40cb-54d3-4c6b-a3d1-27140ece3383 admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:42:07.153371 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-022b411c-2fc4-4b7c-9949-6364411c3447 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:42:07.154812 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-022b411c-2fc4-4b7c-9949-6364411c3447 service cinder] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:42:07.164726 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1311/2614] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 16:42:06 2022] GET /identity/v3/auth/tokens => generated 3258 bytes in 186 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 16:42:07.165346 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-022b411c-2fc4-4b7c-9949-6364411c3447 service cinder] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:42:07.175794 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-022b411c-2fc4-4b7c-9949-6364411c3447 service cinder] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:42:07.180383 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=94,keystone:INSERT=3,keystone:DELETE=3 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:42:07.184996 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-bf1d40cb-54d3-4c6b-a3d1-27140ece3383 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'e4e1f8566af24fcaab73006738c35cce', 'request_id': 'req-bf1d40cb-54d3-4c6b-a3d1-27140ece3383'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:42:07.199198 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-bf1d40cb-54d3-4c6b-a3d1-27140ece3383 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'e4e1f8566af24fcaab73006738c35cce', 'request_id': 'req-bf1d40cb-54d3-4c6b-a3d1-27140ece3383'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:42:07.205967 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-bf1d40cb-54d3-4c6b-a3d1-27140ece3383 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'e4e1f8566af24fcaab73006738c35cce', 'request_id': 'req-bf1d40cb-54d3-4c6b-a3d1-27140ece3383'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:42:07.220877 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-bf1d40cb-54d3-4c6b-a3d1-27140ece3383 admin admin] The token cache is being invalidate because project e4e1f8566af24fcaab73006738c35cce was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81438) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 13 16:42:07.221545 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-bf1d40cb-54d3-4c6b-a3d1-27140ece3383 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:42:07.222049 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-bf1d40cb-54d3-4c6b-a3d1-27140ece3383 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:42:07.223962 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1304/2615] 10.222.0.50 () {58 vars in 1233 bytes} [Tue Sep 13 16:42:07 2022] DELETE /identity/v3/projects/e4e1f8566af24fcaab73006738c35cce => generated 0 bytes in 119 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 16:42:07.227774 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf1d40cb-54d3-4c6b-a3d1-27140ece3383 admin admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:42:07.229512 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-bf1d40cb-54d3-4c6b-a3d1-27140ece3383 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:42:07.240893 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba9abc53-beff-499b-88fa-a2ac97fd211c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:42:07.241904 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba9abc53-beff-499b-88fa-a2ac97fd211c admin admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:42:07.241904 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba9abc53-beff-499b-88fa-a2ac97fd211c admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:42:07.242094 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba9abc53-beff-499b-88fa-a2ac97fd211c admin admin] PATH_INFO: `/v3/roles` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:42:07.242724 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba9abc53-beff-499b-88fa-a2ac97fd211c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:42:07.244548 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba9abc53-beff-499b-88fa-a2ac97fd211c admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:42:07.270454 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1312/2616] 10.222.0.50 () {58 vars in 1122 bytes} [Tue Sep 13 16:42:07 2022] GET /identity/v3/roles => generated 1420 bytes in 103 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 16:42:07.271699 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba9abc53-beff-499b-88fa-a2ac97fd211c admin admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:42:07.274793 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-ba9abc53-beff-499b-88fa-a2ac97fd211c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:42:07.287632 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba9abc53-beff-499b-88fa-a2ac97fd211c admin admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:42:07.292578 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02404ccf-c256-49e0-89c3-19d4a0439caf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:42:07.293179 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02404ccf-c256-49e0-89c3-19d4a0439caf admin admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:42:07.293457 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02404ccf-c256-49e0-89c3-19d4a0439caf admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:42:07.293733 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02404ccf-c256-49e0-89c3-19d4a0439caf admin admin] PATH_INFO: `/v3/roles` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:42:07.294374 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02404ccf-c256-49e0-89c3-19d4a0439caf admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:42:07.296018 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02404ccf-c256-49e0-89c3-19d4a0439caf admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:42:07.306841 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1313/2617] 10.222.0.50 () {58 vars in 1122 bytes} [Tue Sep 13 16:42:07 2022] GET /identity/v3/roles => generated 1420 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 16:42:07.310850 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02404ccf-c256-49e0-89c3-19d4a0439caf admin admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:42:07.317827 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf1d40cb-54d3-4c6b-a3d1-27140ece3383 admin admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:42:07.323690 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02404ccf-c256-49e0-89c3-19d4a0439caf admin admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:42:07.324434 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8338239c-fa05-47cd-ac8d-d20744c985d7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:42:07.325074 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8338239c-fa05-47cd-ac8d-d20744c985d7 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:42:07.325352 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8338239c-fa05-47cd-ac8d-d20744c985d7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:42:07.325621 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8338239c-fa05-47cd-ac8d-d20744c985d7 admin admin] PATH_INFO: `/v3/users/6b7190f35fb94b3489ab34dd77a47c14` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:42:07.327278 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8338239c-fa05-47cd-ac8d-d20744c985d7 admin admin] RBAC: Authorizing `identity:delete_user(user_id=6b7190f35fb94b3489ab34dd77a47c14)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:42:07.329306 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-538e0ed9-0094-4998-878f-970721514239 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:42:07.330385 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-538e0ed9-0094-4998-878f-970721514239 admin admin] REQUEST_METHOD: `PUT` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:42:07.330744 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-538e0ed9-0094-4998-878f-970721514239 admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:42:07.331049 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8338239c-fa05-47cd-ac8d-d20744c985d7 admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:42:07.331160 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-538e0ed9-0094-4998-878f-970721514239 admin admin] PATH_INFO: `/v3/projects/9a74c97db23a45b7bec0def4fa75a119/users/3c3055d03a794d039b9a7f9f7ea5b4c4/roles/15db4aaca3264ae29bde38903ba43471` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:42:07.352501 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-538e0ed9-0094-4998-878f-970721514239 admin admin] RBAC: Authorizing `identity:create_grant(project_id=9a74c97db23a45b7bec0def4fa75a119, user_id=3c3055d03a794d039b9a7f9f7ea5b4c4, role_id=15db4aaca3264ae29bde38903ba43471)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:42:07.355560 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-538e0ed9-0094-4998-878f-970721514239 admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:42:07.375442 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1314/2618] 10.222.0.50 () {60 vars in 1483 bytes} [Tue Sep 13 16:42:07 2022] PUT /identity/v3/projects/9a74c97db23a45b7bec0def4fa75a119/users/3c3055d03a794d039b9a7f9f7ea5b4c4/roles/15db4aaca3264ae29bde38903ba43471 => generated 0 bytes in 64 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 13 16:42:07.387621 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc88130e-2979-4303-9211-04d4e0aa2ecf None None] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:42:07.388057 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc88130e-2979-4303-9211-04d4e0aa2ecf None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:42:07.388410 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc88130e-2979-4303-9211-04d4e0aa2ecf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:42:07.397564 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-8338239c-fa05-47cd-ac8d-d20744c985d7 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '6b7190f35fb94b3489ab34dd77a47c14'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:42:07.409702 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-8338239c-fa05-47cd-ac8d-d20744c985d7 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '6b7190f35fb94b3489ab34dd77a47c14'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:42:07.428020 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-8338239c-fa05-47cd-ac8d-d20744c985d7 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '6b7190f35fb94b3489ab34dd77a47c14'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:42:07.428633 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-8338239c-fa05-47cd-ac8d-d20744c985d7 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '6b7190f35fb94b3489ab34dd77a47c14'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:42:07.434489 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.auth.core [None req-bc88130e-2979-4303-9211-04d4e0aa2ecf None None] MFA Rules not processed for user `3c3055d03a794d039b9a7f9f7ea5b4c4`. Rule list: `[]` (Enabled: `True`). {{(pid=81437) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:42:07.435817 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-bc88130e-2979-4303-9211-04d4e0aa2ecf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:42:07.438156 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-8338239c-fa05-47cd-ac8d-d20744c985d7 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '6b7190f35fb94b3489ab34dd77a47c14'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:42:07.445961 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1305/2619] 10.222.0.50 () {58 vars in 1224 bytes} [Tue Sep 13 16:42:07 2022] DELETE /identity/v3/users/6b7190f35fb94b3489ab34dd77a47c14 => generated 0 bytes in 219 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 13 16:42:07.494965 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:42:07.525862 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1315/2620] 10.222.0.50 () {58 vars in 958 bytes} [Tue Sep 13 16:42:07 2022] POST /identity/v3/auth/tokens => generated 3258 bytes in 136 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:42:07.527163 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8338239c-fa05-47cd-ac8d-d20744c985d7 admin admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:42:07.530634 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-8338239c-fa05-47cd-ac8d-d20744c985d7 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:42:07.624726 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=84,keystone:DELETE=14,keystone:INSERT=2 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:42:07.647417 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8338239c-fa05-47cd-ac8d-d20744c985d7 admin admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:42:07.652428 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b6a45fc-4bba-47ef-bcaf-548a52d94c29 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:42:07.652940 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b6a45fc-4bba-47ef-bcaf-548a52d94c29 admin admin] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:42:07.653228 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b6a45fc-4bba-47ef-bcaf-548a52d94c29 admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:42:07.653510 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b6a45fc-4bba-47ef-bcaf-548a52d94c29 admin admin] PATH_INFO: `/v3/projects` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:42:07.654164 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b6a45fc-4bba-47ef-bcaf-548a52d94c29 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:42:07.656097 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b6a45fc-4bba-47ef-bcaf-548a52d94c29 admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:42:07.690300 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1306/2621] 10.222.0.50 () {60 vars in 1153 bytes} [Tue Sep 13 16:42:07 2022] POST /identity/v3/projects => generated 383 bytes in 160 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 16:42:07.691467 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc88130e-2979-4303-9211-04d4e0aa2ecf None None] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:42:07.766387 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc88130e-2979-4303-9211-04d4e0aa2ecf None None] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:42:07.774780 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b6a45fc-4bba-47ef-bcaf-548a52d94c29 admin admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:42:07.776858 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-2b6a45fc-4bba-47ef-bcaf-548a52d94c29 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:42:07.843037 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f693ba8e-03f2-406c-9878-ef13f01564d3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:42:07.843779 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f693ba8e-03f2-406c-9878-ef13f01564d3 admin admin] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:42:07.844063 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f693ba8e-03f2-406c-9878-ef13f01564d3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:42:07.844340 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f693ba8e-03f2-406c-9878-ef13f01564d3 admin admin] PATH_INFO: `/v3/users` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:42:07.845573 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f693ba8e-03f2-406c-9878-ef13f01564d3 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:42:07.847631 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f693ba8e-03f2-406c-9878-ef13f01564d3 admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:42:07.872614 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1316/2622] 10.222.0.50 () {60 vars in 1144 bytes} [Tue Sep 13 16:42:07 2022] POST /identity/v3/users => generated 298 bytes in 182 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 16:42:07.876666 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f693ba8e-03f2-406c-9878-ef13f01564d3 admin admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:42:07.877637 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b6a45fc-4bba-47ef-bcaf-548a52d94c29 admin admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:42:07.882546 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a67a3041-db59-4904-b6d5-8c11dbec87e3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:42:07.883211 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a67a3041-db59-4904-b6d5-8c11dbec87e3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:42:07.883493 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a67a3041-db59-4904-b6d5-8c11dbec87e3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:42:07.883785 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a67a3041-db59-4904-b6d5-8c11dbec87e3 admin admin] PATH_INFO: `/v3/projects/86830c84bc9b44ae95573dfdff3f5bcf` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:42:07.885034 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a67a3041-db59-4904-b6d5-8c11dbec87e3 admin admin] RBAC: Authorizing `identity:delete_project(project_id=86830c84bc9b44ae95573dfdff3f5bcf)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:42:07.886574 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f693ba8e-03f2-406c-9878-ef13f01564d3 admin admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:42:07.886925 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a67a3041-db59-4904-b6d5-8c11dbec87e3 admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:42:07.891355 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9cc84b0-40fb-4dfe-874d-e63fbd11b48f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:42:07.891946 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9cc84b0-40fb-4dfe-874d-e63fbd11b48f admin admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:42:07.892222 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9cc84b0-40fb-4dfe-874d-e63fbd11b48f admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:42:07.892492 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9cc84b0-40fb-4dfe-874d-e63fbd11b48f admin admin] PATH_INFO: `/v3/roles` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:42:07.893125 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9cc84b0-40fb-4dfe-874d-e63fbd11b48f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:42:07.894725 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9cc84b0-40fb-4dfe-874d-e63fbd11b48f admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:42:07.907551 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1317/2623] 10.222.0.50 () {58 vars in 1122 bytes} [Tue Sep 13 16:42:07 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 16:42:07.909040 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9cc84b0-40fb-4dfe-874d-e63fbd11b48f admin admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:42:07.919435 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9cc84b0-40fb-4dfe-874d-e63fbd11b48f admin admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:42:07.925753 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88f7cf1a-0cfb-4198-87d1-5d88e31066c5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:42:07.926429 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88f7cf1a-0cfb-4198-87d1-5d88e31066c5 admin admin] REQUEST_METHOD: `PUT` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:42:07.926758 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88f7cf1a-0cfb-4198-87d1-5d88e31066c5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:42:07.927072 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88f7cf1a-0cfb-4198-87d1-5d88e31066c5 admin admin] PATH_INFO: `/v3/projects/c079b4d3f9fe4f15a389d81cb2d5e469/users/f1e337e487c445658091806a16cdbdfb/roles/031d4e1a23e34f71858fc4a6dfbd4ccb` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:42:07.939592 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-a67a3041-db59-4904-b6d5-8c11dbec87e3 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '86830c84bc9b44ae95573dfdff3f5bcf', 'request_id': 'req-a67a3041-db59-4904-b6d5-8c11dbec87e3'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:42:07.944518 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88f7cf1a-0cfb-4198-87d1-5d88e31066c5 admin admin] RBAC: Authorizing `identity:create_grant(project_id=c079b4d3f9fe4f15a389d81cb2d5e469, user_id=f1e337e487c445658091806a16cdbdfb, role_id=031d4e1a23e34f71858fc4a6dfbd4ccb)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:42:07.947861 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-a67a3041-db59-4904-b6d5-8c11dbec87e3 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '86830c84bc9b44ae95573dfdff3f5bcf', 'request_id': 'req-a67a3041-db59-4904-b6d5-8c11dbec87e3'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:42:07.948687 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88f7cf1a-0cfb-4198-87d1-5d88e31066c5 admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:42:07.954049 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-a67a3041-db59-4904-b6d5-8c11dbec87e3 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '86830c84bc9b44ae95573dfdff3f5bcf', 'request_id': 'req-a67a3041-db59-4904-b6d5-8c11dbec87e3'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:42:07.962335 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1318/2624] 10.222.0.50 () {60 vars in 1483 bytes} [Tue Sep 13 16:42:07 2022] PUT /identity/v3/projects/c079b4d3f9fe4f15a389d81cb2d5e469/users/f1e337e487c445658091806a16cdbdfb/roles/031d4e1a23e34f71858fc4a6dfbd4ccb => generated 0 bytes in 53 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 16:42:07.965962 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88f7cf1a-0cfb-4198-87d1-5d88e31066c5 admin admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:42:07.966200 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-a67a3041-db59-4904-b6d5-8c11dbec87e3 admin admin] The token cache is being invalidate because project 86830c84bc9b44ae95573dfdff3f5bcf was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81438) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 13 16:42:07.966509 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-a67a3041-db59-4904-b6d5-8c11dbec87e3 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:42:07.967231 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-a67a3041-db59-4904-b6d5-8c11dbec87e3 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:42:07.968886 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1307/2625] 10.222.0.50 () {58 vars in 1233 bytes} [Tue Sep 13 16:42:07 2022] DELETE /identity/v3/projects/86830c84bc9b44ae95573dfdff3f5bcf => generated 0 bytes in 194 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 13 16:42:07.973929 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d131a3c1-ad84-4aac-bfc4-b1cf27fd2d0e None None] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:42:07.974321 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d131a3c1-ad84-4aac-bfc4-b1cf27fd2d0e None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:42:07.974695 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d131a3c1-ad84-4aac-bfc4-b1cf27fd2d0e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:42:07.997011 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.auth.core [None req-d131a3c1-ad84-4aac-bfc4-b1cf27fd2d0e None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81438) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:42:07.998974 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-d131a3c1-ad84-4aac-bfc4-b1cf27fd2d0e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:42:08.020803 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:42:08.037279 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88f7cf1a-0cfb-4198-87d1-5d88e31066c5 admin admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:42:08.042012 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8baaa9cd-dfd9-43ce-9c9e-ad0260c3b5cf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:42:08.042615 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8baaa9cd-dfd9-43ce-9c9e-ad0260c3b5cf admin admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:42:08.042941 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8baaa9cd-dfd9-43ce-9c9e-ad0260c3b5cf admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:42:08.043261 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8baaa9cd-dfd9-43ce-9c9e-ad0260c3b5cf admin admin] PATH_INFO: `/v3/roles` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:42:08.043902 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8baaa9cd-dfd9-43ce-9c9e-ad0260c3b5cf admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:42:08.045509 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8baaa9cd-dfd9-43ce-9c9e-ad0260c3b5cf admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:42:08.055943 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1319/2626] 10.222.0.50 () {58 vars in 1122 bytes} [Tue Sep 13 16:42:07 2022] GET /identity/v3/roles => generated 1420 bytes in 90 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 16:42:08.059680 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8baaa9cd-dfd9-43ce-9c9e-ad0260c3b5cf admin admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:42:08.061403 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-8baaa9cd-dfd9-43ce-9c9e-ad0260c3b5cf admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:42:08.073064 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1308/2627] 10.222.0.50 () {58 vars in 958 bytes} [Tue Sep 13 16:42:07 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 13 16:42:08.074603 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8baaa9cd-dfd9-43ce-9c9e-ad0260c3b5cf admin admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:42:08.080602 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbf8c953-32c5-4d4f-9e48-f383f3f83cb0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:42:08.081346 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbf8c953-32c5-4d4f-9e48-f383f3f83cb0 admin admin] REQUEST_METHOD: `PUT` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:42:08.081627 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbf8c953-32c5-4d4f-9e48-f383f3f83cb0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:42:08.081903 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbf8c953-32c5-4d4f-9e48-f383f3f83cb0 admin admin] PATH_INFO: `/v3/projects/c079b4d3f9fe4f15a389d81cb2d5e469/users/f1e337e487c445658091806a16cdbdfb/roles/031d4e1a23e34f71858fc4a6dfbd4ccb` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:42:08.082002 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-672b7fdf-804a-4234-beb3-8e43fe756137 None None] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:42:08.082278 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-672b7fdf-804a-4234-beb3-8e43fe756137 None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:42:08.082555 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-672b7fdf-804a-4234-beb3-8e43fe756137 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:42:08.083921 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbf8c953-32c5-4d4f-9e48-f383f3f83cb0 admin admin] RBAC: Authorizing `identity:create_grant(project_id=c079b4d3f9fe4f15a389d81cb2d5e469, user_id=f1e337e487c445658091806a16cdbdfb, role_id=031d4e1a23e34f71858fc4a6dfbd4ccb)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:42:08.086591 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbf8c953-32c5-4d4f-9e48-f383f3f83cb0 admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:42:08.097132 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=91,keystone:INSERT=2,keystone:DELETE=7 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:42:08.097784 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1320/2628] 10.222.0.50 () {60 vars in 1483 bytes} [Tue Sep 13 16:42:08 2022] PUT /identity/v3/projects/c079b4d3f9fe4f15a389d81cb2d5e469/users/f1e337e487c445658091806a16cdbdfb/roles/031d4e1a23e34f71858fc4a6dfbd4ccb => generated 0 bytes in 38 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 16:42:08.107298 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.auth.core [None req-672b7fdf-804a-4234-beb3-8e43fe756137 None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81438) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:42:08.107298 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-672b7fdf-804a-4234-beb3-8e43fe756137 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:42:08.108870 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c7072ad-b1c6-43dd-8104-3eccddaa2a25 None None] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:42:08.109188 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c7072ad-b1c6-43dd-8104-3eccddaa2a25 None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:42:08.109469 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c7072ad-b1c6-43dd-8104-3eccddaa2a25 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:42:08.138525 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.auth.core [None req-5c7072ad-b1c6-43dd-8104-3eccddaa2a25 None None] MFA Rules not processed for user `f1e337e487c445658091806a16cdbdfb`. Rule list: `[]` (Enabled: `True`). {{(pid=81437) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:42:08.139761 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-5c7072ad-b1c6-43dd-8104-3eccddaa2a25 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:42:08.177751 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1309/2629] 10.222.0.50 () {58 vars in 958 bytes} [Tue Sep 13 16:42:08 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 13 16:42:08.181942 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-672b7fdf-804a-4234-beb3-8e43fe756137 None None] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:42:08.191593 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-672b7fdf-804a-4234-beb3-8e43fe756137 None None] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:42:08.196436 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdd5a9e8-3919-4931-a0b6-774d84ffcd2f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:42:08.197145 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdd5a9e8-3919-4931-a0b6-774d84ffcd2f admin admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:42:08.197434 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdd5a9e8-3919-4931-a0b6-774d84ffcd2f admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:42:08.197712 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdd5a9e8-3919-4931-a0b6-774d84ffcd2f admin admin] PATH_INFO: `/v3/domains` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:42:08.198360 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdd5a9e8-3919-4931-a0b6-774d84ffcd2f admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81438) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 16:42:08.198667 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdd5a9e8-3919-4931-a0b6-774d84ffcd2f admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:42:08.200311 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdd5a9e8-3919-4931-a0b6-774d84ffcd2f admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:42:08.212896 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1310/2630] 10.222.0.50 () {58 vars in 1153 bytes} [Tue Sep 13 16:42:08 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 13 16:42:08.216689 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1321/2631] 10.222.0.50 () {58 vars in 958 bytes} [Tue Sep 13 16:42:08 2022] POST /identity/v3/auth/tokens => generated 3327 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 13 16:42:08.217064 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdd5a9e8-3919-4931-a0b6-774d84ffcd2f admin admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:42:08.227634 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdd5a9e8-3919-4931-a0b6-774d84ffcd2f admin admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:42:08.228671 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c7072ad-b1c6-43dd-8104-3eccddaa2a25 None None] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:42:08.230031 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-5c7072ad-b1c6-43dd-8104-3eccddaa2a25 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:42:08.232642 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f680ff1-a14a-4761-8f80-1ae78321aab2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:42:08.233235 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f680ff1-a14a-4761-8f80-1ae78321aab2 admin admin] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:42:08.233520 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f680ff1-a14a-4761-8f80-1ae78321aab2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:42:08.233793 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f680ff1-a14a-4761-8f80-1ae78321aab2 admin admin] PATH_INFO: `/v3/projects` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:42:08.234419 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f680ff1-a14a-4761-8f80-1ae78321aab2 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:42:08.236495 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f680ff1-a14a-4761-8f80-1ae78321aab2 admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:42:08.268579 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1311/2632] 10.222.0.50 () {60 vars in 1153 bytes} [Tue Sep 13 16:42:08 2022] POST /identity/v3/projects => generated 359 bytes in 48 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 16:42:08.269548 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f680ff1-a14a-4761-8f80-1ae78321aab2 admin admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:42:08.288334 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c7072ad-b1c6-43dd-8104-3eccddaa2a25 None None] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:42:08.344091 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f680ff1-a14a-4761-8f80-1ae78321aab2 admin admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:42:08.348083 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68e21c79-3931-47e8-abce-5cf703e14f1b service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7fd6b5afd67b48f9b0f0597ccd4cf4df', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:42:08.348747 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68e21c79-3931-47e8-abce-5cf703e14f1b service cinder] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:42:08.349054 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68e21c79-3931-47e8-abce-5cf703e14f1b service cinder] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:42:08.349329 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68e21c79-3931-47e8-abce-5cf703e14f1b service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:42:08.349829 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eece37a3-ee12-484d-91f5-8794f03d27db admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:42:08.350434 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eece37a3-ee12-484d-91f5-8794f03d27db admin admin] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:42:08.350721 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eece37a3-ee12-484d-91f5-8794f03d27db admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:42:08.351031 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eece37a3-ee12-484d-91f5-8794f03d27db admin admin] PATH_INFO: `/v3/users` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:42:08.351197 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-68e21c79-3931-47e8-abce-5cf703e14f1b service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:42:08.351664 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eece37a3-ee12-484d-91f5-8794f03d27db admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:42:08.353590 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eece37a3-ee12-484d-91f5-8794f03d27db admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:42:08.376725 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1312/2633] 10.222.0.50 () {60 vars in 1144 bytes} [Tue Sep 13 16:42:08 2022] POST /identity/v3/users => generated 280 bytes in 108 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 13 16:42:08.380985 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eece37a3-ee12-484d-91f5-8794f03d27db admin admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:42:08.382594 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:42:08.390810 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eece37a3-ee12-484d-91f5-8794f03d27db admin admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:42:08.396500 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a7cf6e5-d0d9-458f-9450-eb43c0b85cd4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:42:08.397105 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a7cf6e5-d0d9-458f-9450-eb43c0b85cd4 admin admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:42:08.397790 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a7cf6e5-d0d9-458f-9450-eb43c0b85cd4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:42:08.397790 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a7cf6e5-d0d9-458f-9450-eb43c0b85cd4 admin admin] PATH_INFO: `/v3/roles` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:42:08.398294 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a7cf6e5-d0d9-458f-9450-eb43c0b85cd4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:42:08.400233 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a7cf6e5-d0d9-458f-9450-eb43c0b85cd4 admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:42:08.410720 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1313/2634] 10.222.0.50 () {58 vars in 1122 bytes} [Tue Sep 13 16:42:08 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 13 16:42:08.414677 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a7cf6e5-d0d9-458f-9450-eb43c0b85cd4 admin admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:42:08.418659 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68e21c79-3931-47e8-abce-5cf703e14f1b service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:42:08.420184 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68e21c79-3931-47e8-abce-5cf703e14f1b service cinder] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:42:08.424849 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a7cf6e5-d0d9-458f-9450-eb43c0b85cd4 admin admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:42:08.429703 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a74e71d0-cec2-4973-ab6f-c8369a7c51f9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:42:08.429865 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1322/2635] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 16:42:08 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 202 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 13 16:42:08.430441 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a74e71d0-cec2-4973-ab6f-c8369a7c51f9 admin admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:42:08.430723 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a74e71d0-cec2-4973-ab6f-c8369a7c51f9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:42:08.431038 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a74e71d0-cec2-4973-ab6f-c8369a7c51f9 admin admin] PATH_INFO: `/v3/roles` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:42:08.431714 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a74e71d0-cec2-4973-ab6f-c8369a7c51f9 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:42:08.433316 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a74e71d0-cec2-4973-ab6f-c8369a7c51f9 admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:42:08.446942 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1314/2636] 10.222.0.50 () {58 vars in 1122 bytes} [Tue Sep 13 16:42:08 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 13 16:42:08.452901 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68e21c79-3931-47e8-abce-5cf703e14f1b service cinder] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:42:08.466762 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68e21c79-3931-47e8-abce-5cf703e14f1b service cinder] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:42:08.471723 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a45fc45-9b9b-4a83-a524-19e604c7e363 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:42:08.472389 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a45fc45-9b9b-4a83-a524-19e604c7e363 admin admin] REQUEST_METHOD: `PUT` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:42:08.472673 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a45fc45-9b9b-4a83-a524-19e604c7e363 admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:42:08.472951 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a45fc45-9b9b-4a83-a524-19e604c7e363 admin admin] PATH_INFO: `/v3/projects/9a67884a4707400da3ad410ca8a4969b/users/f181a9ed70ee41af899852cdcdd7044b/roles/15db4aaca3264ae29bde38903ba43471` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:42:08.488618 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a45fc45-9b9b-4a83-a524-19e604c7e363 admin admin] RBAC: Authorizing `identity:create_grant(project_id=9a67884a4707400da3ad410ca8a4969b, user_id=f181a9ed70ee41af899852cdcdd7044b, role_id=15db4aaca3264ae29bde38903ba43471)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:42:08.491362 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a45fc45-9b9b-4a83-a524-19e604c7e363 admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:42:08.506595 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1323/2637] 10.222.0.50 () {60 vars in 1483 bytes} [Tue Sep 13 16:42:08 2022] PUT /identity/v3/projects/9a67884a4707400da3ad410ca8a4969b/users/f181a9ed70ee41af899852cdcdd7044b/roles/15db4aaca3264ae29bde38903ba43471 => generated 0 bytes in 54 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 16:42:08.518269 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-524d4487-1606-4306-9df6-043177fc9ba7 None None] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:42:08.518559 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-524d4487-1606-4306-9df6-043177fc9ba7 None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:42:08.518839 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-524d4487-1606-4306-9df6-043177fc9ba7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:42:08.548539 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.auth.core [None req-524d4487-1606-4306-9df6-043177fc9ba7 None None] MFA Rules not processed for user `f181a9ed70ee41af899852cdcdd7044b`. Rule list: `[]` (Enabled: `True`). {{(pid=81438) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:42:08.549771 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-524d4487-1606-4306-9df6-043177fc9ba7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:42:08.585171 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:42:08.646221 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1315/2638] 10.222.0.50 () {58 vars in 958 bytes} [Tue Sep 13 16:42:08 2022] POST /identity/v3/auth/tokens => generated 3236 bytes in 127 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:42:08.734160 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a45fc45-9b9b-4a83-a524-19e604c7e363 admin admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:42:08.738081 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-0a45fc45-9b9b-4a83-a524-19e604c7e363 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:42:08.813944 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a45fc45-9b9b-4a83-a524-19e604c7e363 admin admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:42:08.818192 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57d335a9-8087-4cb9-8b16-f9890167a0e4 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7fd6b5afd67b48f9b0f0597ccd4cf4df', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:42:08.818999 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57d335a9-8087-4cb9-8b16-f9890167a0e4 service cinder] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:42:08.819356 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57d335a9-8087-4cb9-8b16-f9890167a0e4 service cinder] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:42:08.820230 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57d335a9-8087-4cb9-8b16-f9890167a0e4 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:42:08.832203 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57d335a9-8087-4cb9-8b16-f9890167a0e4 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:42:08.836712 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57d335a9-8087-4cb9-8b16-f9890167a0e4 service cinder] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:42:08.847286 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1324/2639] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 16:42:08 2022] GET /identity/v3/auth/tokens => generated 3236 bytes in 114 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 16:42:08.865459 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-524d4487-1606-4306-9df6-043177fc9ba7 None None] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:42:08.952448 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-524d4487-1606-4306-9df6-043177fc9ba7 None None] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:42:08.958058 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7603728a-7337-42e1-baf6-3ec64e202bfb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:42:08.959039 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7603728a-7337-42e1-baf6-3ec64e202bfb admin admin] REQUEST_METHOD: `DELETE` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:42:08.959323 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7603728a-7337-42e1-baf6-3ec64e202bfb admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:42:08.959594 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7603728a-7337-42e1-baf6-3ec64e202bfb admin admin] PATH_INFO: `/v3/users/f181a9ed70ee41af899852cdcdd7044b` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:42:08.960729 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7603728a-7337-42e1-baf6-3ec64e202bfb admin admin] RBAC: Authorizing `identity:delete_user(user_id=f181a9ed70ee41af899852cdcdd7044b)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:42:08.962878 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7603728a-7337-42e1-baf6-3ec64e202bfb admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:42:09.025711 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-7603728a-7337-42e1-baf6-3ec64e202bfb admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'f181a9ed70ee41af899852cdcdd7044b'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:42:09.036609 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-7603728a-7337-42e1-baf6-3ec64e202bfb admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'f181a9ed70ee41af899852cdcdd7044b'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:42:09.051913 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-7603728a-7337-42e1-baf6-3ec64e202bfb admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'f181a9ed70ee41af899852cdcdd7044b'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:42:09.052663 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-7603728a-7337-42e1-baf6-3ec64e202bfb admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'f181a9ed70ee41af899852cdcdd7044b'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:42:09.059720 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-7603728a-7337-42e1-baf6-3ec64e202bfb admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'f181a9ed70ee41af899852cdcdd7044b'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:42:09.067094 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1316/2640] 10.222.0.50 () {58 vars in 1224 bytes} [Tue Sep 13 16:42:08 2022] DELETE /identity/v3/users/f181a9ed70ee41af899852cdcdd7044b => generated 0 bytes in 202 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 16:42:09.071864 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae49d443-b66b-45d0-805a-068b28b8ce01 None None] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:42:09.072149 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae49d443-b66b-45d0-805a-068b28b8ce01 None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:42:09.072424 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae49d443-b66b-45d0-805a-068b28b8ce01 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:42:09.093520 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.auth.core [None req-ae49d443-b66b-45d0-805a-068b28b8ce01 None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81437) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:42:09.094769 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-ae49d443-b66b-45d0-805a-068b28b8ce01 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:42:09.162883 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1325/2641] 10.222.0.50 () {58 vars in 958 bytes} [Tue Sep 13 16:42:09 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:42:09.174029 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7603728a-7337-42e1-baf6-3ec64e202bfb admin admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:42:09.175583 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-7603728a-7337-42e1-baf6-3ec64e202bfb admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:42:09.240877 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:42:09.246433 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7603728a-7337-42e1-baf6-3ec64e202bfb admin admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:42:09.253261 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b52616f9-251b-405c-8bc5-20348207aaa1 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f436f07939d4855a21b195bc7566244', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:42:09.253831 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b52616f9-251b-405c-8bc5-20348207aaa1 service neutron] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:42:09.254230 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b52616f9-251b-405c-8bc5-20348207aaa1 service neutron] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:42:09.254727 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b52616f9-251b-405c-8bc5-20348207aaa1 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:42:09.266408 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b52616f9-251b-405c-8bc5-20348207aaa1 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:42:09.268355 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b52616f9-251b-405c-8bc5-20348207aaa1 service neutron] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:42:09.278419 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1317/2642] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 16:42:09 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 106 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 16:42:09.619736 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae49d443-b66b-45d0-805a-068b28b8ce01 None None] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:42:09.621505 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-ae49d443-b66b-45d0-805a-068b28b8ce01 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:42:09.635933 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae49d443-b66b-45d0-805a-068b28b8ce01 None None] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:42:09.641340 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-168a5f28-5dc8-4cf9-a5af-df70657191a4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:42:09.642050 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-168a5f28-5dc8-4cf9-a5af-df70657191a4 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:42:09.642459 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-168a5f28-5dc8-4cf9-a5af-df70657191a4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:42:09.642773 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-168a5f28-5dc8-4cf9-a5af-df70657191a4 admin admin] PATH_INFO: `/v3/projects/9a67884a4707400da3ad410ca8a4969b` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:42:09.645491 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-168a5f28-5dc8-4cf9-a5af-df70657191a4 admin admin] RBAC: Authorizing `identity:delete_project(project_id=9a67884a4707400da3ad410ca8a4969b)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:42:09.648192 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-168a5f28-5dc8-4cf9-a5af-df70657191a4 admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:42:09.698735 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=96,keystone:INSERT=1,keystone:DELETE=3 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:42:09.708261 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-168a5f28-5dc8-4cf9-a5af-df70657191a4 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '9a67884a4707400da3ad410ca8a4969b', 'request_id': 'req-168a5f28-5dc8-4cf9-a5af-df70657191a4'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:42:09.720574 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-168a5f28-5dc8-4cf9-a5af-df70657191a4 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '9a67884a4707400da3ad410ca8a4969b', 'request_id': 'req-168a5f28-5dc8-4cf9-a5af-df70657191a4'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:42:09.727748 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-168a5f28-5dc8-4cf9-a5af-df70657191a4 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '9a67884a4707400da3ad410ca8a4969b', 'request_id': 'req-168a5f28-5dc8-4cf9-a5af-df70657191a4'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:42:09.741248 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-168a5f28-5dc8-4cf9-a5af-df70657191a4 admin admin] The token cache is being invalidate because project 9a67884a4707400da3ad410ca8a4969b was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81437) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 13 16:42:09.741620 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-168a5f28-5dc8-4cf9-a5af-df70657191a4 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:42:09.748040 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-168a5f28-5dc8-4cf9-a5af-df70657191a4 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:42:09.749590 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1326/2643] 10.222.0.50 () {58 vars in 1233 bytes} [Tue Sep 13 16:42:09 2022] DELETE /identity/v3/projects/9a67884a4707400da3ad410ca8a4969b => generated 0 bytes in 131 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 16:42:09.755152 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e12c8968-cc9b-4d42-9e15-1ddd584e30e5 None None] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:42:09.755602 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e12c8968-cc9b-4d42-9e15-1ddd584e30e5 None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:42:09.755712 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e12c8968-cc9b-4d42-9e15-1ddd584e30e5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:42:09.775033 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.auth.core [None req-e12c8968-cc9b-4d42-9e15-1ddd584e30e5 None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81438) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:42:09.776243 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-e12c8968-cc9b-4d42-9e15-1ddd584e30e5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:42:09.840841 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1318/2644] 10.222.0.50 () {58 vars in 958 bytes} [Tue Sep 13 16:42:09 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:42:09.849929 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d1a5298-05b0-4f6e-91ab-ff5f641638b1 None None] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:42:09.850370 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d1a5298-05b0-4f6e-91ab-ff5f641638b1 None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:42:09.850750 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d1a5298-05b0-4f6e-91ab-ff5f641638b1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:42:09.872668 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.auth.core [None req-3d1a5298-05b0-4f6e-91ab-ff5f641638b1 None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81437) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:42:09.874072 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-3d1a5298-05b0-4f6e-91ab-ff5f641638b1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:42:09.883851 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1327/2645] 10.222.0.50 () {58 vars in 958 bytes} [Tue Sep 13 16:42:09 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 36 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:42:09.888581 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e12c8968-cc9b-4d42-9e15-1ddd584e30e5 None None] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:42:09.900968 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e12c8968-cc9b-4d42-9e15-1ddd584e30e5 None None] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:42:09.906555 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77c5a3dc-22e3-4e21-9559-519d9b06187e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:42:09.907933 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77c5a3dc-22e3-4e21-9559-519d9b06187e admin admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:42:09.908285 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77c5a3dc-22e3-4e21-9559-519d9b06187e admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:42:09.908568 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77c5a3dc-22e3-4e21-9559-519d9b06187e admin admin] PATH_INFO: `/v3/domains` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:42:09.909226 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77c5a3dc-22e3-4e21-9559-519d9b06187e admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81438) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 16:42:09.909536 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77c5a3dc-22e3-4e21-9559-519d9b06187e admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:42:09.911180 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77c5a3dc-22e3-4e21-9559-519d9b06187e admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:42:09.926421 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1319/2646] 10.222.0.50 () {58 vars in 1153 bytes} [Tue Sep 13 16:42:09 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 16:42:09.930830 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d1a5298-05b0-4f6e-91ab-ff5f641638b1 None None] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:42:09.942147 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d1a5298-05b0-4f6e-91ab-ff5f641638b1 None None] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:42:09.956129 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f12c3659-cbb5-48df-b795-120301225224 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:42:09.956856 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f12c3659-cbb5-48df-b795-120301225224 admin admin] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:42:09.957202 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f12c3659-cbb5-48df-b795-120301225224 admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:42:09.957557 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f12c3659-cbb5-48df-b795-120301225224 admin admin] PATH_INFO: `/v3/projects` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:42:09.958322 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f12c3659-cbb5-48df-b795-120301225224 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:42:09.960454 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f12c3659-cbb5-48df-b795-120301225224 admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:42:09.971857 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77c5a3dc-22e3-4e21-9559-519d9b06187e admin admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:42:09.975748 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-77c5a3dc-22e3-4e21-9559-519d9b06187e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:42:09.990067 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77c5a3dc-22e3-4e21-9559-519d9b06187e admin admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:42:09.998191 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-311c7c01-a1ea-48c0-8e17-45d7c7761705 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:42:09.998191 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-311c7c01-a1ea-48c0-8e17-45d7c7761705 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:42:09.998191 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-311c7c01-a1ea-48c0-8e17-45d7c7761705 admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:42:09.998191 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-311c7c01-a1ea-48c0-8e17-45d7c7761705 admin admin] PATH_INFO: `/v3/users/0e2becf5a7704a7f9aca6e1ae6f7f340` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:42:09.998807 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1328/2647] 10.222.0.50 () {60 vars in 1153 bytes} [Tue Sep 13 16:42:09 2022] POST /identity/v3/projects => generated 385 bytes in 65 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 13 16:42:09.999736 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f12c3659-cbb5-48df-b795-120301225224 admin admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:42:10.005018 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-311c7c01-a1ea-48c0-8e17-45d7c7761705 admin admin] RBAC: Authorizing `identity:delete_user(user_id=0e2becf5a7704a7f9aca6e1ae6f7f340)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:42:10.006986 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-311c7c01-a1ea-48c0-8e17-45d7c7761705 admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:42:10.063394 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-311c7c01-a1ea-48c0-8e17-45d7c7761705 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '0e2becf5a7704a7f9aca6e1ae6f7f340'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:42:10.067810 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-311c7c01-a1ea-48c0-8e17-45d7c7761705 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '0e2becf5a7704a7f9aca6e1ae6f7f340'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:42:10.071915 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f12c3659-cbb5-48df-b795-120301225224 admin admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:42:10.076799 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0160351-5aea-4a66-bcda-e0f0f42ea98d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:42:10.077415 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0160351-5aea-4a66-bcda-e0f0f42ea98d admin admin] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:42:10.077694 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0160351-5aea-4a66-bcda-e0f0f42ea98d admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:42:10.077970 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0160351-5aea-4a66-bcda-e0f0f42ea98d admin admin] PATH_INFO: `/v3/users` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:42:10.078603 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0160351-5aea-4a66-bcda-e0f0f42ea98d admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:42:10.080837 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0160351-5aea-4a66-bcda-e0f0f42ea98d admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:42:10.081013 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-311c7c01-a1ea-48c0-8e17-45d7c7761705 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '0e2becf5a7704a7f9aca6e1ae6f7f340'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:42:10.081705 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-311c7c01-a1ea-48c0-8e17-45d7c7761705 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '0e2becf5a7704a7f9aca6e1ae6f7f340'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:42:10.088713 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-311c7c01-a1ea-48c0-8e17-45d7c7761705 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '0e2becf5a7704a7f9aca6e1ae6f7f340'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:42:10.095943 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1320/2648] 10.222.0.50 () {58 vars in 1224 bytes} [Tue Sep 13 16:42:09 2022] DELETE /identity/v3/users/0e2becf5a7704a7f9aca6e1ae6f7f340 => generated 0 bytes in 125 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 13 16:42:10.099734 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-311c7c01-a1ea-48c0-8e17-45d7c7761705 admin admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:42:10.101170 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-311c7c01-a1ea-48c0-8e17-45d7c7761705 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:42:10.102942 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1329/2649] 10.222.0.50 () {60 vars in 1144 bytes} [Tue Sep 13 16:42:09 2022] POST /identity/v3/users => generated 293 bytes in 103 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 16:42:10.104050 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bcbe8f6-a033-4733-b558-295d6a194e88 None None] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:42:10.104347 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bcbe8f6-a033-4733-b558-295d6a194e88 None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:42:10.104613 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bcbe8f6-a033-4733-b558-295d6a194e88 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:42:10.124448 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.auth.core [None req-3bcbe8f6-a033-4733-b558-295d6a194e88 None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81437) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:42:10.125792 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-3bcbe8f6-a033-4733-b558-295d6a194e88 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:42:10.167782 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-311c7c01-a1ea-48c0-8e17-45d7c7761705 admin admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:42:10.171474 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af7da580-1b48-4140-abc0-32174af46617 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7fd6b5afd67b48f9b0f0597ccd4cf4df', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:42:10.172096 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af7da580-1b48-4140-abc0-32174af46617 service cinder] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:42:10.172367 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af7da580-1b48-4140-abc0-32174af46617 service cinder] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:42:10.173335 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af7da580-1b48-4140-abc0-32174af46617 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:42:10.175205 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-af7da580-1b48-4140-abc0-32174af46617 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:42:10.180397 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:42:10.187251 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1330/2650] 10.222.0.50 () {58 vars in 958 bytes} [Tue Sep 13 16:42:10 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:42:10.188387 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bcbe8f6-a033-4733-b558-295d6a194e88 None None] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:42:10.189765 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-3bcbe8f6-a033-4733-b558-295d6a194e88 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:42:10.201932 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bcbe8f6-a033-4733-b558-295d6a194e88 None None] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:42:10.207102 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0476965a-6d7c-4dab-bf7a-539c73d297b6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:42:10.207713 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0476965a-6d7c-4dab-bf7a-539c73d297b6 admin admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:42:10.207990 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0476965a-6d7c-4dab-bf7a-539c73d297b6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:42:10.208262 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0476965a-6d7c-4dab-bf7a-539c73d297b6 admin admin] PATH_INFO: `/v3/roles` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:42:10.208934 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0476965a-6d7c-4dab-bf7a-539c73d297b6 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:42:10.210535 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0476965a-6d7c-4dab-bf7a-539c73d297b6 admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:42:10.221377 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1331/2651] 10.222.0.50 () {58 vars in 1122 bytes} [Tue Sep 13 16:42:10 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 16:42:10.222180 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0476965a-6d7c-4dab-bf7a-539c73d297b6 admin admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:42:10.224141 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-0476965a-6d7c-4dab-bf7a-539c73d297b6 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:42:10.241561 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af7da580-1b48-4140-abc0-32174af46617 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:42:10.243026 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af7da580-1b48-4140-abc0-32174af46617 service cinder] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:42:10.246552 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=91,keystone:DELETE=4,keystone:INSERT=5 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:42:10.252951 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1321/2652] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 16:42:10 2022] GET /identity/v3/auth/tokens => generated 3258 bytes in 156 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 16:42:10.253768 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af7da580-1b48-4140-abc0-32174af46617 service cinder] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:42:10.264843 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af7da580-1b48-4140-abc0-32174af46617 service cinder] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:42:10.272672 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c99aae37-ae8b-4fe8-918e-73552d631d49 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:42:10.273281 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c99aae37-ae8b-4fe8-918e-73552d631d49 admin admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:42:10.273564 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c99aae37-ae8b-4fe8-918e-73552d631d49 admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:42:10.273841 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c99aae37-ae8b-4fe8-918e-73552d631d49 admin admin] PATH_INFO: `/v3/roles` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:42:10.274500 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c99aae37-ae8b-4fe8-918e-73552d631d49 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:42:10.276138 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c99aae37-ae8b-4fe8-918e-73552d631d49 admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:42:10.287243 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1322/2653] 10.222.0.50 () {58 vars in 1122 bytes} [Tue Sep 13 16:42:10 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 16:42:10.290982 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0476965a-6d7c-4dab-bf7a-539c73d297b6 admin admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:42:10.291895 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c99aae37-ae8b-4fe8-918e-73552d631d49 admin admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:42:10.294731 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c4653bd-ed3a-46ea-a5ee-a0309e146988 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f436f07939d4855a21b195bc7566244', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:42:10.295365 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c4653bd-ed3a-46ea-a5ee-a0309e146988 service neutron] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:42:10.295642 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c4653bd-ed3a-46ea-a5ee-a0309e146988 service neutron] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:42:10.295910 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c4653bd-ed3a-46ea-a5ee-a0309e146988 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:42:10.302446 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c99aae37-ae8b-4fe8-918e-73552d631d49 admin admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:42:10.307777 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c4653bd-ed3a-46ea-a5ee-a0309e146988 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:42:10.308603 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0728e827-7bfd-404e-b196-505f401448eb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:42:10.309400 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0728e827-7bfd-404e-b196-505f401448eb admin admin] REQUEST_METHOD: `PUT` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:42:10.309535 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c4653bd-ed3a-46ea-a5ee-a0309e146988 service neutron] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:42:10.309938 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0728e827-7bfd-404e-b196-505f401448eb admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:42:10.310536 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0728e827-7bfd-404e-b196-505f401448eb admin admin] PATH_INFO: `/v3/projects/a655f5436deb42789ba6bc2364d50a64/users/2853507d50984ac8b51990fde7bd8029/roles/15db4aaca3264ae29bde38903ba43471` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:42:10.321632 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1332/2654] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 16:42:10 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 99 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 16:42:10.329201 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0728e827-7bfd-404e-b196-505f401448eb admin admin] RBAC: Authorizing `identity:create_grant(project_id=a655f5436deb42789ba6bc2364d50a64, user_id=2853507d50984ac8b51990fde7bd8029, role_id=15db4aaca3264ae29bde38903ba43471)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:42:10.332395 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0728e827-7bfd-404e-b196-505f401448eb admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:42:10.354981 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1323/2655] 10.222.0.50 () {60 vars in 1483 bytes} [Tue Sep 13 16:42:10 2022] PUT /identity/v3/projects/a655f5436deb42789ba6bc2364d50a64/users/2853507d50984ac8b51990fde7bd8029/roles/15db4aaca3264ae29bde38903ba43471 => generated 0 bytes in 58 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 13 16:42:10.361765 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5842dfdb-03aa-4c93-8831-1af731d3ee1a None None] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:42:10.362184 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5842dfdb-03aa-4c93-8831-1af731d3ee1a None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:42:10.362555 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5842dfdb-03aa-4c93-8831-1af731d3ee1a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:42:10.395093 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.auth.core [None req-5842dfdb-03aa-4c93-8831-1af731d3ee1a None None] MFA Rules not processed for user `2853507d50984ac8b51990fde7bd8029`. Rule list: `[]` (Enabled: `True`). {{(pid=81437) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:42:10.396461 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-5842dfdb-03aa-4c93-8831-1af731d3ee1a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:42:10.478083 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1333/2656] 10.222.0.50 () {58 vars in 958 bytes} [Tue Sep 13 16:42:10 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 119 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 13 16:42:10.494659 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0728e827-7bfd-404e-b196-505f401448eb admin admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:42:10.497096 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-0728e827-7bfd-404e-b196-505f401448eb admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:42:10.571424 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0728e827-7bfd-404e-b196-505f401448eb admin admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:42:10.575913 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-973ff5a6-b86d-4a44-a3af-2579534d814b service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7fd6b5afd67b48f9b0f0597ccd4cf4df', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:42:10.576558 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-973ff5a6-b86d-4a44-a3af-2579534d814b service cinder] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:42:10.576837 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-973ff5a6-b86d-4a44-a3af-2579534d814b service cinder] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:42:10.577122 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-973ff5a6-b86d-4a44-a3af-2579534d814b service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:42:10.586406 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-973ff5a6-b86d-4a44-a3af-2579534d814b service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:42:10.587886 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-973ff5a6-b86d-4a44-a3af-2579534d814b service cinder] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:42:10.597366 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1324/2657] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 16:42:10 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 103 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 16:42:10.656059 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5842dfdb-03aa-4c93-8831-1af731d3ee1a None None] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:42:10.658353 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-5842dfdb-03aa-4c93-8831-1af731d3ee1a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:42:10.756278 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5842dfdb-03aa-4c93-8831-1af731d3ee1a None None] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:42:10.763812 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a54087f1-6280-4872-bb86-7a50580ed591 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:42:10.764597 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a54087f1-6280-4872-bb86-7a50580ed591 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:42:10.764877 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a54087f1-6280-4872-bb86-7a50580ed591 admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:42:10.765163 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a54087f1-6280-4872-bb86-7a50580ed591 admin admin] PATH_INFO: `/v3/projects/54b78d5cc7e9452f9694afe2bb869174` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:42:10.766796 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a54087f1-6280-4872-bb86-7a50580ed591 admin admin] RBAC: Authorizing `identity:delete_project(project_id=54b78d5cc7e9452f9694afe2bb869174)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:42:10.769176 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a54087f1-6280-4872-bb86-7a50580ed591 admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:42:10.825274 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-a54087f1-6280-4872-bb86-7a50580ed591 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '54b78d5cc7e9452f9694afe2bb869174', 'request_id': 'req-a54087f1-6280-4872-bb86-7a50580ed591'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:42:10.830014 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=94,keystone:DELETE=5,keystone:INSERT=1 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:42:10.836670 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-a54087f1-6280-4872-bb86-7a50580ed591 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '54b78d5cc7e9452f9694afe2bb869174', 'request_id': 'req-a54087f1-6280-4872-bb86-7a50580ed591'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:42:10.851782 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-a54087f1-6280-4872-bb86-7a50580ed591 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '54b78d5cc7e9452f9694afe2bb869174', 'request_id': 'req-a54087f1-6280-4872-bb86-7a50580ed591'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:42:10.865069 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-a54087f1-6280-4872-bb86-7a50580ed591 admin admin] The token cache is being invalidate because project 54b78d5cc7e9452f9694afe2bb869174 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81437) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 13 16:42:10.865361 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-a54087f1-6280-4872-bb86-7a50580ed591 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:42:10.866084 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-a54087f1-6280-4872-bb86-7a50580ed591 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:42:10.867897 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1334/2658] 10.222.0.50 () {58 vars in 1233 bytes} [Tue Sep 13 16:42:10 2022] DELETE /identity/v3/projects/54b78d5cc7e9452f9694afe2bb869174 => generated 0 bytes in 213 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 16:42:10.872036 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-973ff5a6-b86d-4a44-a3af-2579534d814b service cinder] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:42:10.899586 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-973ff5a6-b86d-4a44-a3af-2579534d814b service cinder] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:42:10.966065 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a54087f1-6280-4872-bb86-7a50580ed591 admin admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:42:10.967766 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-a54087f1-6280-4872-bb86-7a50580ed591 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:42:11.053995 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a54087f1-6280-4872-bb86-7a50580ed591 admin admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:42:11.058504 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8186b3e8-0cc5-4c9d-a677-a80813bf8b0d service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7fd6b5afd67b48f9b0f0597ccd4cf4df', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:42:11.059298 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8186b3e8-0cc5-4c9d-a677-a80813bf8b0d service cinder] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:42:11.059685 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8186b3e8-0cc5-4c9d-a677-a80813bf8b0d service cinder] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:42:11.060063 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8186b3e8-0cc5-4c9d-a677-a80813bf8b0d service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:42:11.062038 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-8186b3e8-0cc5-4c9d-a677-a80813bf8b0d service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:42:11.112710 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0270f342-b8db-4724-ab51-0106ddb0eb6b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:42:11.114086 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0270f342-b8db-4724-ab51-0106ddb0eb6b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:42:11.114363 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0270f342-b8db-4724-ab51-0106ddb0eb6b admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:42:11.114755 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0270f342-b8db-4724-ab51-0106ddb0eb6b admin admin] PATH_INFO: `/v3/users/16e0c1dffbe54278863f506e11d59f36` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:42:11.116044 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0270f342-b8db-4724-ab51-0106ddb0eb6b admin admin] RBAC: Authorizing `identity:delete_user(user_id=16e0c1dffbe54278863f506e11d59f36)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:42:11.118074 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0270f342-b8db-4724-ab51-0106ddb0eb6b admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:42:11.127868 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8186b3e8-0cc5-4c9d-a677-a80813bf8b0d service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:42:11.129297 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8186b3e8-0cc5-4c9d-a677-a80813bf8b0d service cinder] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:42:11.132935 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=98,keystone:INSERT=1,keystone:DELETE=1 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:42:11.140352 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1335/2659] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 16:42:10 2022] GET /identity/v3/auth/tokens => generated 3327 bytes in 175 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 16:42:11.175411 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-0270f342-b8db-4724-ab51-0106ddb0eb6b admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '16e0c1dffbe54278863f506e11d59f36'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:42:11.183869 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-0270f342-b8db-4724-ab51-0106ddb0eb6b admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '16e0c1dffbe54278863f506e11d59f36'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:42:11.197515 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-0270f342-b8db-4724-ab51-0106ddb0eb6b admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '16e0c1dffbe54278863f506e11d59f36'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:42:11.198225 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-0270f342-b8db-4724-ab51-0106ddb0eb6b admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '16e0c1dffbe54278863f506e11d59f36'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:42:11.208762 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-0270f342-b8db-4724-ab51-0106ddb0eb6b admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '16e0c1dffbe54278863f506e11d59f36'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:42:11.223224 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1325/2660] 10.222.0.50 () {58 vars in 1224 bytes} [Tue Sep 13 16:42:10 2022] DELETE /identity/v3/users/16e0c1dffbe54278863f506e11d59f36 => generated 0 bytes in 347 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 16:42:11.496706 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8186b3e8-0cc5-4c9d-a677-a80813bf8b0d service cinder] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:42:11.498314 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-8186b3e8-0cc5-4c9d-a677-a80813bf8b0d service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:42:11.514081 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8186b3e8-0cc5-4c9d-a677-a80813bf8b0d service cinder] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:42:11.576157 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e6fb795-4868-44e1-83b1-4e34ef901e6c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:42:11.576847 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e6fb795-4868-44e1-83b1-4e34ef901e6c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:42:11.577328 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e6fb795-4868-44e1-83b1-4e34ef901e6c admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:42:11.577639 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e6fb795-4868-44e1-83b1-4e34ef901e6c admin admin] PATH_INFO: `/v3/projects/e400cf6cc595413cb1b18435e94fd903` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:42:11.579784 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e6fb795-4868-44e1-83b1-4e34ef901e6c admin admin] RBAC: Authorizing `identity:delete_project(project_id=e400cf6cc595413cb1b18435e94fd903)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:42:11.582493 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e6fb795-4868-44e1-83b1-4e34ef901e6c admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:42:11.621893 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=96,keystone:DELETE=4 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:42:11.633234 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-5e6fb795-4868-44e1-83b1-4e34ef901e6c admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'e400cf6cc595413cb1b18435e94fd903', 'request_id': 'req-5e6fb795-4868-44e1-83b1-4e34ef901e6c'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:42:11.639846 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-5e6fb795-4868-44e1-83b1-4e34ef901e6c admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'e400cf6cc595413cb1b18435e94fd903', 'request_id': 'req-5e6fb795-4868-44e1-83b1-4e34ef901e6c'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:42:11.646355 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-5e6fb795-4868-44e1-83b1-4e34ef901e6c admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'e400cf6cc595413cb1b18435e94fd903', 'request_id': 'req-5e6fb795-4868-44e1-83b1-4e34ef901e6c'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:42:11.657579 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-5e6fb795-4868-44e1-83b1-4e34ef901e6c admin admin] The token cache is being invalidate because project e400cf6cc595413cb1b18435e94fd903 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81437) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 13 16:42:11.658149 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-5e6fb795-4868-44e1-83b1-4e34ef901e6c admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:42:11.658494 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-5e6fb795-4868-44e1-83b1-4e34ef901e6c admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:42:11.660017 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1336/2661] 10.222.0.50 () {58 vars in 1233 bytes} [Tue Sep 13 16:42:11 2022] DELETE /identity/v3/projects/e400cf6cc595413cb1b18435e94fd903 => generated 0 bytes in 164 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 16:42:11.664876 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0c54498-5377-4c0d-900c-de1f2afcf208 None None] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:42:11.665180 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0c54498-5377-4c0d-900c-de1f2afcf208 None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:42:11.665454 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0c54498-5377-4c0d-900c-de1f2afcf208 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:42:11.685159 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.auth.core [None req-c0c54498-5377-4c0d-900c-de1f2afcf208 None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81438) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:42:11.686354 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-c0c54498-5377-4c0d-900c-de1f2afcf208 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:42:11.744000 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1326/2662] 10.222.0.50 () {58 vars in 958 bytes} [Tue Sep 13 16:42:11 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 80 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:42:11.752045 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e812c7c-3190-4b38-a56b-4bf7fb68464d None None] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:42:11.752334 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e812c7c-3190-4b38-a56b-4bf7fb68464d None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:42:11.752611 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e812c7c-3190-4b38-a56b-4bf7fb68464d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:42:11.773454 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.auth.core [None req-4e812c7c-3190-4b38-a56b-4bf7fb68464d None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81437) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:42:11.774665 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-4e812c7c-3190-4b38-a56b-4bf7fb68464d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:42:11.783593 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1337/2663] 10.222.0.50 () {58 vars in 958 bytes} [Tue Sep 13 16:42:11 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:42:11.787775 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0c54498-5377-4c0d-900c-de1f2afcf208 None None] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:42:11.797500 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0c54498-5377-4c0d-900c-de1f2afcf208 None None] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:42:11.802146 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fe36d57-4eda-4a3b-8adf-fccc39834df7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:42:11.802778 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fe36d57-4eda-4a3b-8adf-fccc39834df7 admin admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:42:11.803104 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fe36d57-4eda-4a3b-8adf-fccc39834df7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:42:11.803382 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fe36d57-4eda-4a3b-8adf-fccc39834df7 admin admin] PATH_INFO: `/v3/domains` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:42:11.804023 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fe36d57-4eda-4a3b-8adf-fccc39834df7 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81438) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 16:42:11.804328 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fe36d57-4eda-4a3b-8adf-fccc39834df7 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:42:11.805916 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fe36d57-4eda-4a3b-8adf-fccc39834df7 admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:42:11.817866 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1327/2664] 10.222.0.50 () {58 vars in 1153 bytes} [Tue Sep 13 16:42:11 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 16:42:11.822253 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e812c7c-3190-4b38-a56b-4bf7fb68464d None None] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:42:11.832584 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e812c7c-3190-4b38-a56b-4bf7fb68464d None None] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:42:11.837624 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1688007c-a84c-4e15-9147-a36b7105b7ef admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:42:11.838230 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1688007c-a84c-4e15-9147-a36b7105b7ef admin admin] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:42:11.838510 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1688007c-a84c-4e15-9147-a36b7105b7ef admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:42:11.838782 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1688007c-a84c-4e15-9147-a36b7105b7ef admin admin] PATH_INFO: `/v3/projects` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:42:11.839453 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1688007c-a84c-4e15-9147-a36b7105b7ef admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:42:11.841342 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1688007c-a84c-4e15-9147-a36b7105b7ef admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:42:11.871073 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1338/2665] 10.222.0.50 () {60 vars in 1153 bytes} [Tue Sep 13 16:42:11 2022] POST /identity/v3/projects => generated 383 bytes in 49 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 16:42:11.875446 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fe36d57-4eda-4a3b-8adf-fccc39834df7 admin admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:42:11.885345 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fe36d57-4eda-4a3b-8adf-fccc39834df7 admin admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:42:11.941447 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f50eae3-031d-4bad-aa4d-525cd72ced99 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:42:11.942083 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f50eae3-031d-4bad-aa4d-525cd72ced99 admin admin] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:42:11.942437 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f50eae3-031d-4bad-aa4d-525cd72ced99 admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:42:11.942805 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f50eae3-031d-4bad-aa4d-525cd72ced99 admin admin] PATH_INFO: `/v3/users` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:42:11.943577 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f50eae3-031d-4bad-aa4d-525cd72ced99 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:42:11.945681 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f50eae3-031d-4bad-aa4d-525cd72ced99 admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:42:11.972094 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1328/2666] 10.222.0.50 () {60 vars in 1144 bytes} [Tue Sep 13 16:42:11 2022] POST /identity/v3/users => generated 292 bytes in 94 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 16:42:11.973835 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1688007c-a84c-4e15-9147-a36b7105b7ef admin admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:42:11.985126 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1688007c-a84c-4e15-9147-a36b7105b7ef admin admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:42:11.990512 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eed1d839-4155-4e2b-9051-b99dad862338 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:42:11.991222 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eed1d839-4155-4e2b-9051-b99dad862338 admin admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:42:11.991566 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eed1d839-4155-4e2b-9051-b99dad862338 admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:42:11.991903 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eed1d839-4155-4e2b-9051-b99dad862338 admin admin] PATH_INFO: `/v3/roles` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:42:11.992623 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eed1d839-4155-4e2b-9051-b99dad862338 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:42:11.994282 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eed1d839-4155-4e2b-9051-b99dad862338 admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:42:12.005192 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1339/2667] 10.222.0.50 () {58 vars in 1122 bytes} [Tue Sep 13 16:42:11 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 16:42:12.009106 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f50eae3-031d-4bad-aa4d-525cd72ced99 admin admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:42:12.019946 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f50eae3-031d-4bad-aa4d-525cd72ced99 admin admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:42:12.026650 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd12fc83-33b0-46ab-b7cb-40d9479e6903 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:42:12.027324 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd12fc83-33b0-46ab-b7cb-40d9479e6903 admin admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:42:12.027605 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd12fc83-33b0-46ab-b7cb-40d9479e6903 admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:42:12.027879 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd12fc83-33b0-46ab-b7cb-40d9479e6903 admin admin] PATH_INFO: `/v3/roles` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:42:12.028597 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd12fc83-33b0-46ab-b7cb-40d9479e6903 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:42:12.030078 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd12fc83-33b0-46ab-b7cb-40d9479e6903 admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:42:12.033117 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=87,keystone:DELETE=9,keystone:INSERT=4 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:42:12.056834 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1329/2668] 10.222.0.50 () {58 vars in 1122 bytes} [Tue Sep 13 16:42:12 2022] GET /identity/v3/roles => generated 1420 bytes in 45 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 16:42:12.058032 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eed1d839-4155-4e2b-9051-b99dad862338 admin admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:42:12.072437 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eed1d839-4155-4e2b-9051-b99dad862338 admin admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:42:12.077258 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42905f60-c696-4f59-b779-bd91fd78a18e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:42:12.077911 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42905f60-c696-4f59-b779-bd91fd78a18e admin admin] REQUEST_METHOD: `PUT` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:42:12.078193 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42905f60-c696-4f59-b779-bd91fd78a18e admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:42:12.078466 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42905f60-c696-4f59-b779-bd91fd78a18e admin admin] PATH_INFO: `/v3/projects/b88d4caa3c384949a8f918729576f4ab/users/1f179c0cd3614d67915953617492e862/roles/15db4aaca3264ae29bde38903ba43471` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:42:12.096180 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42905f60-c696-4f59-b779-bd91fd78a18e admin admin] RBAC: Authorizing `identity:create_grant(project_id=b88d4caa3c384949a8f918729576f4ab, user_id=1f179c0cd3614d67915953617492e862, role_id=15db4aaca3264ae29bde38903ba43471)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:42:12.098827 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42905f60-c696-4f59-b779-bd91fd78a18e admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:42:12.117725 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1340/2669] 10.222.0.50 () {60 vars in 1483 bytes} [Tue Sep 13 16:42:12 2022] PUT /identity/v3/projects/b88d4caa3c384949a8f918729576f4ab/users/1f179c0cd3614d67915953617492e862/roles/15db4aaca3264ae29bde38903ba43471 => generated 0 bytes in 60 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 13 16:42:12.129808 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7086ddb4-d306-47e2-80dd-e55b79fbf683 None None] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:42:12.130106 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7086ddb4-d306-47e2-80dd-e55b79fbf683 None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:42:12.130372 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7086ddb4-d306-47e2-80dd-e55b79fbf683 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:42:12.159669 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.auth.core [None req-7086ddb4-d306-47e2-80dd-e55b79fbf683 None None] MFA Rules not processed for user `1f179c0cd3614d67915953617492e862`. Rule list: `[]` (Enabled: `True`). {{(pid=81438) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:42:12.160893 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-7086ddb4-d306-47e2-80dd-e55b79fbf683 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:42:12.231675 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1330/2670] 10.222.0.50 () {58 vars in 958 bytes} [Tue Sep 13 16:42:12 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:42:12.243360 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42905f60-c696-4f59-b779-bd91fd78a18e admin admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:42:12.245083 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-42905f60-c696-4f59-b779-bd91fd78a18e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:42:12.308000 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42905f60-c696-4f59-b779-bd91fd78a18e admin admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:42:12.311779 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9dbf90ed-9fb9-4090-a0b2-cf8aaa6548a0 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7fd6b5afd67b48f9b0f0597ccd4cf4df', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:42:12.312387 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dbf90ed-9fb9-4090-a0b2-cf8aaa6548a0 service cinder] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:42:12.312668 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dbf90ed-9fb9-4090-a0b2-cf8aaa6548a0 service cinder] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:42:12.312943 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dbf90ed-9fb9-4090-a0b2-cf8aaa6548a0 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:42:12.322880 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9dbf90ed-9fb9-4090-a0b2-cf8aaa6548a0 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:42:12.324442 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9dbf90ed-9fb9-4090-a0b2-cf8aaa6548a0 service cinder] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:42:12.335062 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1341/2671] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 16:42:12 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 92 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 16:42:12.355517 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7086ddb4-d306-47e2-80dd-e55b79fbf683 None None] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:42:12.429329 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7086ddb4-d306-47e2-80dd-e55b79fbf683 None None] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:42:12.433973 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e0405dd-66d2-454a-9fcf-e4e660666ec3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:42:12.434603 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e0405dd-66d2-454a-9fcf-e4e660666ec3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:42:12.434887 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e0405dd-66d2-454a-9fcf-e4e660666ec3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:42:12.435226 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e0405dd-66d2-454a-9fcf-e4e660666ec3 admin admin] PATH_INFO: `/v3/users/1f179c0cd3614d67915953617492e862` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:42:12.436383 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e0405dd-66d2-454a-9fcf-e4e660666ec3 admin admin] RBAC: Authorizing `identity:delete_user(user_id=1f179c0cd3614d67915953617492e862)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:42:12.439007 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e0405dd-66d2-454a-9fcf-e4e660666ec3 admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:42:12.488628 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-9e0405dd-66d2-454a-9fcf-e4e660666ec3 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '1f179c0cd3614d67915953617492e862'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:42:12.496291 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-9e0405dd-66d2-454a-9fcf-e4e660666ec3 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '1f179c0cd3614d67915953617492e862'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:42:12.520751 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-9e0405dd-66d2-454a-9fcf-e4e660666ec3 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '1f179c0cd3614d67915953617492e862'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:42:12.521545 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-9e0405dd-66d2-454a-9fcf-e4e660666ec3 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '1f179c0cd3614d67915953617492e862'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:42:12.528841 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-9e0405dd-66d2-454a-9fcf-e4e660666ec3 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '1f179c0cd3614d67915953617492e862'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:42:12.535017 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1331/2672] 10.222.0.50 () {58 vars in 1224 bytes} [Tue Sep 13 16:42:12 2022] DELETE /identity/v3/users/1f179c0cd3614d67915953617492e862 => generated 0 bytes in 180 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 16:42:12.540443 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4eccf84d-ba17-476d-86df-02e3dd45318b None None] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:42:12.541097 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4eccf84d-ba17-476d-86df-02e3dd45318b None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:42:12.541618 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4eccf84d-ba17-476d-86df-02e3dd45318b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:42:12.566012 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.auth.core [None req-4eccf84d-ba17-476d-86df-02e3dd45318b None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81437) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:42:12.567568 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-4eccf84d-ba17-476d-86df-02e3dd45318b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:42:12.636881 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1342/2673] 10.222.0.50 () {58 vars in 958 bytes} [Tue Sep 13 16:42:12 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:42:12.647849 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e0405dd-66d2-454a-9fcf-e4e660666ec3 admin admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:42:12.649189 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-9e0405dd-66d2-454a-9fcf-e4e660666ec3 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:42:12.660112 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:42:12.709757 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e0405dd-66d2-454a-9fcf-e4e660666ec3 admin admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:42:12.713320 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5043a0b9-b31c-4dd4-99a6-0805c4bc0e1e service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f436f07939d4855a21b195bc7566244', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:42:12.713913 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5043a0b9-b31c-4dd4-99a6-0805c4bc0e1e service neutron] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:42:12.714189 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5043a0b9-b31c-4dd4-99a6-0805c4bc0e1e service neutron] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:42:12.714458 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5043a0b9-b31c-4dd4-99a6-0805c4bc0e1e service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:42:12.723834 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5043a0b9-b31c-4dd4-99a6-0805c4bc0e1e service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:42:12.725233 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5043a0b9-b31c-4dd4-99a6-0805c4bc0e1e service neutron] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:42:12.734285 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1332/2674] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 16:42:12 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 87 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 16:42:13.001133 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4eccf84d-ba17-476d-86df-02e3dd45318b None None] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:42:13.002611 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-4eccf84d-ba17-476d-86df-02e3dd45318b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:42:13.009878 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:DELETE=5,keystone:SELECT=92,keystone:INSERT=3 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:42:13.015962 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4eccf84d-ba17-476d-86df-02e3dd45318b None None] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:42:13.022341 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8db1d16-e953-440e-b3ef-e297253ab63a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:42:13.023079 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8db1d16-e953-440e-b3ef-e297253ab63a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:42:13.023308 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8db1d16-e953-440e-b3ef-e297253ab63a admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:42:13.023607 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8db1d16-e953-440e-b3ef-e297253ab63a admin admin] PATH_INFO: `/v3/projects/b88d4caa3c384949a8f918729576f4ab` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:42:13.024743 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8db1d16-e953-440e-b3ef-e297253ab63a admin admin] RBAC: Authorizing `identity:delete_project(project_id=b88d4caa3c384949a8f918729576f4ab)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:42:13.026671 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8db1d16-e953-440e-b3ef-e297253ab63a admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:42:13.074484 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-f8db1d16-e953-440e-b3ef-e297253ab63a admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'b88d4caa3c384949a8f918729576f4ab', 'request_id': 'req-f8db1d16-e953-440e-b3ef-e297253ab63a'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:42:13.082632 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-f8db1d16-e953-440e-b3ef-e297253ab63a admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'b88d4caa3c384949a8f918729576f4ab', 'request_id': 'req-f8db1d16-e953-440e-b3ef-e297253ab63a'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:42:13.089113 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-f8db1d16-e953-440e-b3ef-e297253ab63a admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'b88d4caa3c384949a8f918729576f4ab', 'request_id': 'req-f8db1d16-e953-440e-b3ef-e297253ab63a'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:42:13.099800 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-f8db1d16-e953-440e-b3ef-e297253ab63a admin admin] The token cache is being invalidate because project b88d4caa3c384949a8f918729576f4ab was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81437) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 13 16:42:13.100092 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-f8db1d16-e953-440e-b3ef-e297253ab63a admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:42:13.100733 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-f8db1d16-e953-440e-b3ef-e297253ab63a admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:42:13.102229 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1343/2675] 10.222.0.50 () {58 vars in 1233 bytes} [Tue Sep 13 16:42:12 2022] DELETE /identity/v3/projects/b88d4caa3c384949a8f918729576f4ab => generated 0 bytes in 102 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 16:42:13.107122 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5675f49b-aa9f-40ca-8495-e222710cfe18 None None] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:42:13.107543 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5675f49b-aa9f-40ca-8495-e222710cfe18 None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:42:13.107660 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5675f49b-aa9f-40ca-8495-e222710cfe18 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:42:13.129165 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.auth.core [None req-5675f49b-aa9f-40ca-8495-e222710cfe18 None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81438) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:42:13.130368 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-5675f49b-aa9f-40ca-8495-e222710cfe18 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:42:13.192115 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1333/2676] 10.222.0.50 () {58 vars in 958 bytes} [Tue Sep 13 16:42:13 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:42:13.199904 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8c2b69a-a63c-4f97-acd3-79d2d62e566f None None] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:42:13.200195 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8c2b69a-a63c-4f97-acd3-79d2d62e566f None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:42:13.200474 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8c2b69a-a63c-4f97-acd3-79d2d62e566f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:42:13.221377 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.auth.core [None req-f8c2b69a-a63c-4f97-acd3-79d2d62e566f None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81437) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:42:13.222571 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-f8c2b69a-a63c-4f97-acd3-79d2d62e566f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:42:13.230608 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1344/2677] 10.222.0.50 () {58 vars in 958 bytes} [Tue Sep 13 16:42:13 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:42:13.235508 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5675f49b-aa9f-40ca-8495-e222710cfe18 None None] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:42:13.242433 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8c2b69a-a63c-4f97-acd3-79d2d62e566f None None] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:42:13.243870 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-f8c2b69a-a63c-4f97-acd3-79d2d62e566f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:42:13.245310 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5675f49b-aa9f-40ca-8495-e222710cfe18 None None] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:42:13.249922 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b60e6657-248a-4245-bb9f-d18d41cce836 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:42:13.250611 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b60e6657-248a-4245-bb9f-d18d41cce836 admin admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:42:13.250930 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b60e6657-248a-4245-bb9f-d18d41cce836 admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:42:13.251382 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b60e6657-248a-4245-bb9f-d18d41cce836 admin admin] PATH_INFO: `/v3/domains` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:42:13.252036 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b60e6657-248a-4245-bb9f-d18d41cce836 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81438) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 16:42:13.252340 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b60e6657-248a-4245-bb9f-d18d41cce836 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:42:13.253916 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b60e6657-248a-4245-bb9f-d18d41cce836 admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:42:13.255647 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8c2b69a-a63c-4f97-acd3-79d2d62e566f None None] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:42:13.260634 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43e77684-f7bb-4053-996b-3f6935dbecb7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:42:13.261252 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43e77684-f7bb-4053-996b-3f6935dbecb7 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:42:13.261533 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43e77684-f7bb-4053-996b-3f6935dbecb7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:42:13.261807 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43e77684-f7bb-4053-996b-3f6935dbecb7 admin admin] PATH_INFO: `/v3/users/2853507d50984ac8b51990fde7bd8029` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:42:13.262962 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43e77684-f7bb-4053-996b-3f6935dbecb7 admin admin] RBAC: Authorizing `identity:delete_user(user_id=2853507d50984ac8b51990fde7bd8029)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:42:13.264902 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43e77684-f7bb-4053-996b-3f6935dbecb7 admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:42:13.265463 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1334/2678] 10.222.0.50 () {58 vars in 1153 bytes} [Tue Sep 13 16:42:13 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 16:42:13.270352 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b60e6657-248a-4245-bb9f-d18d41cce836 admin admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:42:13.279939 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b60e6657-248a-4245-bb9f-d18d41cce836 admin admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:42:13.285404 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-304eccbf-cd1d-40f6-b3e6-dce049b28d3c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:42:13.285998 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-304eccbf-cd1d-40f6-b3e6-dce049b28d3c admin admin] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:42:13.286279 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-304eccbf-cd1d-40f6-b3e6-dce049b28d3c admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:42:13.286636 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-304eccbf-cd1d-40f6-b3e6-dce049b28d3c admin admin] PATH_INFO: `/v3/projects` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:42:13.287219 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-304eccbf-cd1d-40f6-b3e6-dce049b28d3c admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:42:13.289092 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-304eccbf-cd1d-40f6-b3e6-dce049b28d3c admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:42:13.314126 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-43e77684-f7bb-4053-996b-3f6935dbecb7 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '2853507d50984ac8b51990fde7bd8029'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:42:13.316879 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1335/2679] 10.222.0.50 () {60 vars in 1153 bytes} [Tue Sep 13 16:42:13 2022] POST /identity/v3/projects => generated 375 bytes in 47 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 16:42:13.321435 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-304eccbf-cd1d-40f6-b3e6-dce049b28d3c admin admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:42:13.322109 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-43e77684-f7bb-4053-996b-3f6935dbecb7 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '2853507d50984ac8b51990fde7bd8029'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:42:13.335421 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-43e77684-f7bb-4053-996b-3f6935dbecb7 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '2853507d50984ac8b51990fde7bd8029'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:42:13.336087 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-43e77684-f7bb-4053-996b-3f6935dbecb7 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '2853507d50984ac8b51990fde7bd8029'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:42:13.343484 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-43e77684-f7bb-4053-996b-3f6935dbecb7 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '2853507d50984ac8b51990fde7bd8029'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:42:13.350091 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1345/2680] 10.222.0.50 () {58 vars in 1224 bytes} [Tue Sep 13 16:42:13 2022] DELETE /identity/v3/users/2853507d50984ac8b51990fde7bd8029 => generated 0 bytes in 108 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 13 16:42:13.354336 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35bb48ec-2b20-480a-81ed-92dc114f916f None None] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:42:13.354620 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35bb48ec-2b20-480a-81ed-92dc114f916f None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:42:13.354892 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35bb48ec-2b20-480a-81ed-92dc114f916f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:42:13.374030 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.auth.core [None req-35bb48ec-2b20-480a-81ed-92dc114f916f None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81437) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:42:13.375251 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-35bb48ec-2b20-480a-81ed-92dc114f916f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:42:13.386215 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-304eccbf-cd1d-40f6-b3e6-dce049b28d3c admin admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:42:13.406200 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:42:13.420443 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=81,keystone:DELETE=17,keystone:INSERT=2 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:42:13.436198 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1346/2681] 10.222.0.50 () {58 vars in 958 bytes} [Tue Sep 13 16:42:13 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 13 16:42:13.446041 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35bb48ec-2b20-480a-81ed-92dc114f916f None None] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:42:13.446819 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf0d397c-5634-4de2-a935-2d9412b0a38f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:42:13.447332 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-35bb48ec-2b20-480a-81ed-92dc114f916f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:42:13.447493 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf0d397c-5634-4de2-a935-2d9412b0a38f admin admin] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:42:13.447772 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf0d397c-5634-4de2-a935-2d9412b0a38f admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:42:13.448046 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf0d397c-5634-4de2-a935-2d9412b0a38f admin admin] PATH_INFO: `/v3/users` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:42:13.448671 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf0d397c-5634-4de2-a935-2d9412b0a38f admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:42:13.450623 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf0d397c-5634-4de2-a935-2d9412b0a38f admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:42:13.474142 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1336/2682] 10.222.0.50 () {60 vars in 1144 bytes} [Tue Sep 13 16:42:13 2022] POST /identity/v3/users => generated 288 bytes in 153 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 13 16:42:13.478053 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf0d397c-5634-4de2-a935-2d9412b0a38f admin admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:42:13.479723 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-bf0d397c-5634-4de2-a935-2d9412b0a38f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:42:13.491085 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf0d397c-5634-4de2-a935-2d9412b0a38f admin admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:42:13.496068 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-467816d8-f90b-4c08-b726-7e45b4b3b5f1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:42:13.496669 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-467816d8-f90b-4c08-b726-7e45b4b3b5f1 admin admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:42:13.496946 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-467816d8-f90b-4c08-b726-7e45b4b3b5f1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:42:13.497213 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-467816d8-f90b-4c08-b726-7e45b4b3b5f1 admin admin] PATH_INFO: `/v3/roles` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:42:13.497843 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-467816d8-f90b-4c08-b726-7e45b4b3b5f1 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:42:13.499464 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-467816d8-f90b-4c08-b726-7e45b4b3b5f1 admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:42:13.507273 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35bb48ec-2b20-480a-81ed-92dc114f916f None None] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:42:13.509518 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1337/2683] 10.222.0.50 () {58 vars in 1122 bytes} [Tue Sep 13 16:42:13 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 13 16:42:13.513988 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-467816d8-f90b-4c08-b726-7e45b4b3b5f1 admin admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:42:13.514083 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e82dbbd-0311-4e15-9bcc-97fe3b901b86 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f436f07939d4855a21b195bc7566244', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:42:13.514450 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e82dbbd-0311-4e15-9bcc-97fe3b901b86 service neutron] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:42:13.514727 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e82dbbd-0311-4e15-9bcc-97fe3b901b86 service neutron] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:42:13.515049 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e82dbbd-0311-4e15-9bcc-97fe3b901b86 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:42:13.524068 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-467816d8-f90b-4c08-b726-7e45b4b3b5f1 admin admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:42:13.525165 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e82dbbd-0311-4e15-9bcc-97fe3b901b86 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:42:13.526613 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e82dbbd-0311-4e15-9bcc-97fe3b901b86 service neutron] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:42:13.528861 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5afb5cfd-49fc-4b05-b88f-2b87b341145c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:42:13.529450 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5afb5cfd-49fc-4b05-b88f-2b87b341145c admin admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:42:13.529732 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5afb5cfd-49fc-4b05-b88f-2b87b341145c admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:42:13.530008 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5afb5cfd-49fc-4b05-b88f-2b87b341145c admin admin] PATH_INFO: `/v3/roles` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:42:13.530682 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5afb5cfd-49fc-4b05-b88f-2b87b341145c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:42:13.533097 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5afb5cfd-49fc-4b05-b88f-2b87b341145c admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:42:13.537458 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1347/2684] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 16:42:13 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 92 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 13 16:42:13.543973 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1338/2685] 10.222.0.50 () {58 vars in 1122 bytes} [Tue Sep 13 16:42:13 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 13 16:42:13.548600 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e82dbbd-0311-4e15-9bcc-97fe3b901b86 service neutron] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:42:13.558314 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e82dbbd-0311-4e15-9bcc-97fe3b901b86 service neutron] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:42:13.563887 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a2b271f-c541-4272-a1d0-14810969da56 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:42:13.564600 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a2b271f-c541-4272-a1d0-14810969da56 admin admin] REQUEST_METHOD: `PUT` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:42:13.564965 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a2b271f-c541-4272-a1d0-14810969da56 admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:42:13.565404 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a2b271f-c541-4272-a1d0-14810969da56 admin admin] PATH_INFO: `/v3/projects/395df427f71c403c8617c8aa5592b254/users/79b893555a1849e7901df3a2061ac370/roles/15db4aaca3264ae29bde38903ba43471` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:42:13.580906 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a2b271f-c541-4272-a1d0-14810969da56 admin admin] RBAC: Authorizing `identity:create_grant(project_id=395df427f71c403c8617c8aa5592b254, user_id=79b893555a1849e7901df3a2061ac370, role_id=15db4aaca3264ae29bde38903ba43471)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:42:13.583616 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a2b271f-c541-4272-a1d0-14810969da56 admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:42:13.610385 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1348/2686] 10.222.0.50 () {60 vars in 1483 bytes} [Tue Sep 13 16:42:13 2022] PUT /identity/v3/projects/395df427f71c403c8617c8aa5592b254/users/79b893555a1849e7901df3a2061ac370/roles/15db4aaca3264ae29bde38903ba43471 => generated 0 bytes in 53 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 13 16:42:13.612152 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1069393-c3ec-44d1-9dd0-04866327f59a None None] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:42:13.612484 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1069393-c3ec-44d1-9dd0-04866327f59a None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:42:13.612757 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1069393-c3ec-44d1-9dd0-04866327f59a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:42:13.654807 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.auth.core [None req-e1069393-c3ec-44d1-9dd0-04866327f59a None None] MFA Rules not processed for user `79b893555a1849e7901df3a2061ac370`. Rule list: `[]` (Enabled: `True`). {{(pid=81438) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:42:13.656202 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-e1069393-c3ec-44d1-9dd0-04866327f59a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:42:13.740988 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1339/2687] 10.222.0.50 () {58 vars in 958 bytes} [Tue Sep 13 16:42:13 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 130 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:42:13.755095 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a2b271f-c541-4272-a1d0-14810969da56 admin admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:42:13.755427 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-9a2b271f-c541-4272-a1d0-14810969da56 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:42:13.840630 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a2b271f-c541-4272-a1d0-14810969da56 admin admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:42:13.842040 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1069393-c3ec-44d1-9dd0-04866327f59a None None] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:42:13.843684 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-e1069393-c3ec-44d1-9dd0-04866327f59a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:42:13.845758 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47ab770d-1d27-4120-a23d-c089e1aa7810 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7fd6b5afd67b48f9b0f0597ccd4cf4df', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:42:13.846522 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47ab770d-1d27-4120-a23d-c089e1aa7810 service cinder] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:42:13.846868 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47ab770d-1d27-4120-a23d-c089e1aa7810 service cinder] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:42:13.847270 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47ab770d-1d27-4120-a23d-c089e1aa7810 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:42:13.857703 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47ab770d-1d27-4120-a23d-c089e1aa7810 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:42:13.859421 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47ab770d-1d27-4120-a23d-c089e1aa7810 service cinder] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:42:13.871339 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1349/2688] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 16:42:13 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 118 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 16:42:13.938937 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1069393-c3ec-44d1-9dd0-04866327f59a None None] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:42:13.944261 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f3004b0-0cc6-4668-919c-a8b1d27ebdd5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:42:13.944895 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f3004b0-0cc6-4668-919c-a8b1d27ebdd5 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:42:13.945169 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f3004b0-0cc6-4668-919c-a8b1d27ebdd5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:42:13.945436 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f3004b0-0cc6-4668-919c-a8b1d27ebdd5 admin admin] PATH_INFO: `/v3/projects/a655f5436deb42789ba6bc2364d50a64` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:42:13.946743 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f3004b0-0cc6-4668-919c-a8b1d27ebdd5 admin admin] RBAC: Authorizing `identity:delete_project(project_id=a655f5436deb42789ba6bc2364d50a64)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:42:13.948836 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f3004b0-0cc6-4668-919c-a8b1d27ebdd5 admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:42:13.978240 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:42:14.007185 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-6f3004b0-0cc6-4668-919c-a8b1d27ebdd5 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'a655f5436deb42789ba6bc2364d50a64', 'request_id': 'req-6f3004b0-0cc6-4668-919c-a8b1d27ebdd5'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:42:14.017048 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-6f3004b0-0cc6-4668-919c-a8b1d27ebdd5 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'a655f5436deb42789ba6bc2364d50a64', 'request_id': 'req-6f3004b0-0cc6-4668-919c-a8b1d27ebdd5'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:42:14.024144 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-6f3004b0-0cc6-4668-919c-a8b1d27ebdd5 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'a655f5436deb42789ba6bc2364d50a64', 'request_id': 'req-6f3004b0-0cc6-4668-919c-a8b1d27ebdd5'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:42:14.040502 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-6f3004b0-0cc6-4668-919c-a8b1d27ebdd5 admin admin] The token cache is being invalidate because project a655f5436deb42789ba6bc2364d50a64 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81438) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 13 16:42:14.040770 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-6f3004b0-0cc6-4668-919c-a8b1d27ebdd5 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:42:14.041447 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-6f3004b0-0cc6-4668-919c-a8b1d27ebdd5 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:42:14.043153 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1340/2689] 10.222.0.50 () {58 vars in 1233 bytes} [Tue Sep 13 16:42:13 2022] DELETE /identity/v3/projects/a655f5436deb42789ba6bc2364d50a64 => generated 0 bytes in 201 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 13 16:42:14.048589 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d745e785-3110-4140-a6a7-f833763d6f80 None None] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:42:14.049125 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d745e785-3110-4140-a6a7-f833763d6f80 None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:42:14.049125 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d745e785-3110-4140-a6a7-f833763d6f80 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:42:14.077908 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.auth.core [None req-d745e785-3110-4140-a6a7-f833763d6f80 None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81437) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:42:14.079444 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-d745e785-3110-4140-a6a7-f833763d6f80 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:42:14.150478 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1350/2690] 10.222.0.50 () {58 vars in 958 bytes} [Tue Sep 13 16:42:14 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:42:14.160556 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-055bb9fa-c89d-409c-8e85-7e5d8519d30b None None] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:42:14.161131 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-055bb9fa-c89d-409c-8e85-7e5d8519d30b None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:42:14.161243 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-055bb9fa-c89d-409c-8e85-7e5d8519d30b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:42:14.185986 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.auth.core [None req-055bb9fa-c89d-409c-8e85-7e5d8519d30b None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81438) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:42:14.187481 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-055bb9fa-c89d-409c-8e85-7e5d8519d30b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:42:14.197568 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1341/2691] 10.222.0.50 () {58 vars in 958 bytes} [Tue Sep 13 16:42:14 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 38 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 13 16:42:14.202180 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d745e785-3110-4140-a6a7-f833763d6f80 None None] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:42:14.213509 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d745e785-3110-4140-a6a7-f833763d6f80 None None] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:42:14.223854 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e0a0a1e-83b4-4894-a38c-a021ef6c6c6d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:42:14.224555 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e0a0a1e-83b4-4894-a38c-a021ef6c6c6d admin admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:42:14.224798 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e0a0a1e-83b4-4894-a38c-a021ef6c6c6d admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:42:14.225072 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e0a0a1e-83b4-4894-a38c-a021ef6c6c6d admin admin] PATH_INFO: `/v3/domains` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:42:14.225898 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e0a0a1e-83b4-4894-a38c-a021ef6c6c6d admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81437) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 16:42:14.226203 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e0a0a1e-83b4-4894-a38c-a021ef6c6c6d admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:42:14.228297 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e0a0a1e-83b4-4894-a38c-a021ef6c6c6d admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:42:14.242292 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:42:14.248307 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1351/2692] 10.222.0.50 () {58 vars in 1153 bytes} [Tue Sep 13 16:42:14 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 46 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 16:42:14.253892 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-055bb9fa-c89d-409c-8e85-7e5d8519d30b None None] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:42:14.264503 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-055bb9fa-c89d-409c-8e85-7e5d8519d30b None None] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:42:14.270212 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-695fe02d-9efe-4b15-a638-c47b0768d901 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:42:14.270761 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-695fe02d-9efe-4b15-a638-c47b0768d901 admin admin] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:42:14.271085 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-695fe02d-9efe-4b15-a638-c47b0768d901 admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:42:14.271365 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-695fe02d-9efe-4b15-a638-c47b0768d901 admin admin] PATH_INFO: `/v3/projects` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:42:14.272093 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-695fe02d-9efe-4b15-a638-c47b0768d901 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:42:14.273973 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-695fe02d-9efe-4b15-a638-c47b0768d901 admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:42:14.310098 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1342/2693] 10.222.0.50 () {60 vars in 1153 bytes} [Tue Sep 13 16:42:14 2022] POST /identity/v3/projects => generated 397 bytes in 57 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 16:42:14.315121 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e0a0a1e-83b4-4894-a38c-a021ef6c6c6d admin admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:42:14.324877 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e0a0a1e-83b4-4894-a38c-a021ef6c6c6d admin admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:42:14.385253 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b72f2ae4-0e87-4cb1-b3e1-b2895fe88336 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:42:14.385857 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b72f2ae4-0e87-4cb1-b3e1-b2895fe88336 admin admin] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:42:14.386136 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b72f2ae4-0e87-4cb1-b3e1-b2895fe88336 admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:42:14.386411 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b72f2ae4-0e87-4cb1-b3e1-b2895fe88336 admin admin] PATH_INFO: `/v3/users` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:42:14.387067 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b72f2ae4-0e87-4cb1-b3e1-b2895fe88336 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:42:14.388990 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b72f2ae4-0e87-4cb1-b3e1-b2895fe88336 admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:42:14.411287 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1352/2694] 10.222.0.50 () {60 vars in 1144 bytes} [Tue Sep 13 16:42:14 2022] POST /identity/v3/users => generated 299 bytes in 97 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 16:42:14.415562 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-695fe02d-9efe-4b15-a638-c47b0768d901 admin admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:42:14.426655 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-695fe02d-9efe-4b15-a638-c47b0768d901 admin admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:42:14.431318 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e8d41cd-3f52-4853-abae-5bf9ffd9fdc4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:42:14.431914 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e8d41cd-3f52-4853-abae-5bf9ffd9fdc4 admin admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:42:14.432194 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e8d41cd-3f52-4853-abae-5bf9ffd9fdc4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:42:14.432470 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e8d41cd-3f52-4853-abae-5bf9ffd9fdc4 admin admin] PATH_INFO: `/v3/roles` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:42:14.433106 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e8d41cd-3f52-4853-abae-5bf9ffd9fdc4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:42:14.434692 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e8d41cd-3f52-4853-abae-5bf9ffd9fdc4 admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:42:14.445522 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1343/2695] 10.222.0.50 () {58 vars in 1122 bytes} [Tue Sep 13 16:42:14 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 16:42:14.449798 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b72f2ae4-0e87-4cb1-b3e1-b2895fe88336 admin admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:42:14.461395 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b72f2ae4-0e87-4cb1-b3e1-b2895fe88336 admin admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:42:14.466218 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3f18a24-390e-4cab-9a5d-af468f85921f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:42:14.466816 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3f18a24-390e-4cab-9a5d-af468f85921f admin admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:42:14.467124 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3f18a24-390e-4cab-9a5d-af468f85921f admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:42:14.467403 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3f18a24-390e-4cab-9a5d-af468f85921f admin admin] PATH_INFO: `/v3/roles` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:42:14.468039 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3f18a24-390e-4cab-9a5d-af468f85921f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:42:14.469950 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3f18a24-390e-4cab-9a5d-af468f85921f admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:42:14.482098 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1353/2696] 10.222.0.50 () {58 vars in 1122 bytes} [Tue Sep 13 16:42:14 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 16:42:14.486558 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e8d41cd-3f52-4853-abae-5bf9ffd9fdc4 admin admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:42:14.496541 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e8d41cd-3f52-4853-abae-5bf9ffd9fdc4 admin admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:42:14.501773 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be173f67-f8a6-4d8d-aad6-bf4d872b20e9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:42:14.502417 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be173f67-f8a6-4d8d-aad6-bf4d872b20e9 admin admin] REQUEST_METHOD: `PUT` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:42:14.502697 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be173f67-f8a6-4d8d-aad6-bf4d872b20e9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:42:14.503010 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be173f67-f8a6-4d8d-aad6-bf4d872b20e9 admin admin] PATH_INFO: `/v3/projects/414f0a9cc70f45ec9c9dfc55964392c0/users/3c322eaf6cee42b0a24faf6fb61d227d/roles/15db4aaca3264ae29bde38903ba43471` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:42:14.518994 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be173f67-f8a6-4d8d-aad6-bf4d872b20e9 admin admin] RBAC: Authorizing `identity:create_grant(project_id=414f0a9cc70f45ec9c9dfc55964392c0, user_id=3c322eaf6cee42b0a24faf6fb61d227d, role_id=15db4aaca3264ae29bde38903ba43471)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:42:14.521686 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be173f67-f8a6-4d8d-aad6-bf4d872b20e9 admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:42:14.534506 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1344/2697] 10.222.0.50 () {60 vars in 1483 bytes} [Tue Sep 13 16:42:14 2022] PUT /identity/v3/projects/414f0a9cc70f45ec9c9dfc55964392c0/users/3c322eaf6cee42b0a24faf6fb61d227d/roles/15db4aaca3264ae29bde38903ba43471 => generated 0 bytes in 48 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 13 16:42:14.545199 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e12782c9-e716-4b66-b11f-d859a38065c7 None None] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:42:14.545948 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e12782c9-e716-4b66-b11f-d859a38065c7 None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:42:14.545948 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e12782c9-e716-4b66-b11f-d859a38065c7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:42:14.576001 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.auth.core [None req-e12782c9-e716-4b66-b11f-d859a38065c7 None None] MFA Rules not processed for user `3c322eaf6cee42b0a24faf6fb61d227d`. Rule list: `[]` (Enabled: `True`). {{(pid=81437) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:42:14.577218 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-e12782c9-e716-4b66-b11f-d859a38065c7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:42:14.640577 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1354/2698] 10.222.0.50 () {58 vars in 958 bytes} [Tue Sep 13 16:42:14 2022] POST /identity/v3/auth/tokens => generated 3274 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:42:14.652849 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be173f67-f8a6-4d8d-aad6-bf4d872b20e9 admin admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:42:14.654502 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-be173f67-f8a6-4d8d-aad6-bf4d872b20e9 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:42:14.710676 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be173f67-f8a6-4d8d-aad6-bf4d872b20e9 admin admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:42:14.714184 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d453836-4a2a-41cd-8bd0-d1a7ef402335 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7fd6b5afd67b48f9b0f0597ccd4cf4df', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:42:14.714778 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d453836-4a2a-41cd-8bd0-d1a7ef402335 service cinder] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:42:14.715100 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d453836-4a2a-41cd-8bd0-d1a7ef402335 service cinder] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:42:14.715370 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d453836-4a2a-41cd-8bd0-d1a7ef402335 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:42:14.723543 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d453836-4a2a-41cd-8bd0-d1a7ef402335 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:42:14.724943 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d453836-4a2a-41cd-8bd0-d1a7ef402335 service cinder] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:42:14.733444 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1345/2699] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 16:42:14 2022] GET /identity/v3/auth/tokens => generated 3274 bytes in 81 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 16:42:15.108431 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf843b7f-1b45-44bc-a148-d98565e50c7f None None] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:42:15.108930 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf843b7f-1b45-44bc-a148-d98565e50c7f None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:42:15.109043 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf843b7f-1b45-44bc-a148-d98565e50c7f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:42:15.134618 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.auth.core [None req-bf843b7f-1b45-44bc-a148-d98565e50c7f None None] MFA Rules not processed for user `b6417e189b0d467480627cfd38331a9f`. Rule list: `[]` (Enabled: `True`). {{(pid=81437) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:42:15.136409 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-bf843b7f-1b45-44bc-a148-d98565e50c7f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:42:15.221427 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d453836-4a2a-41cd-8bd0-d1a7ef402335 service cinder] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:42:15.223053 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-5d453836-4a2a-41cd-8bd0-d1a7ef402335 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:42:15.236464 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1355/2700] 10.222.0.50 () {58 vars in 958 bytes} [Tue Sep 13 16:42:15 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 125 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:42:15.314321 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:DELETE=7,keystone:SELECT=90,keystone:INSERT=3 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:42:15.337437 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d453836-4a2a-41cd-8bd0-d1a7ef402335 service cinder] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:42:15.342036 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-120dbc1b-b810-49b6-a0ae-8831021d8e02 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:42:15.342813 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-120dbc1b-b810-49b6-a0ae-8831021d8e02 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:42:15.343133 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-120dbc1b-b810-49b6-a0ae-8831021d8e02 admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:42:15.343406 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-120dbc1b-b810-49b6-a0ae-8831021d8e02 admin admin] PATH_INFO: `/v3/users/c3c5559749a54790a5bf891c827e0ced` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:42:15.345111 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-120dbc1b-b810-49b6-a0ae-8831021d8e02 admin admin] RBAC: Authorizing `identity:delete_user(user_id=c3c5559749a54790a5bf891c827e0ced)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:42:15.347203 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-120dbc1b-b810-49b6-a0ae-8831021d8e02 admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:42:15.400479 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-120dbc1b-b810-49b6-a0ae-8831021d8e02 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'c3c5559749a54790a5bf891c827e0ced'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:42:15.408287 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-120dbc1b-b810-49b6-a0ae-8831021d8e02 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'c3c5559749a54790a5bf891c827e0ced'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:42:15.423427 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-120dbc1b-b810-49b6-a0ae-8831021d8e02 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'c3c5559749a54790a5bf891c827e0ced'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:42:15.424645 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-120dbc1b-b810-49b6-a0ae-8831021d8e02 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'c3c5559749a54790a5bf891c827e0ced'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:42:15.433365 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-120dbc1b-b810-49b6-a0ae-8831021d8e02 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'c3c5559749a54790a5bf891c827e0ced'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:42:15.442155 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1346/2701] 10.222.0.50 () {58 vars in 1224 bytes} [Tue Sep 13 16:42:15 2022] DELETE /identity/v3/users/c3c5559749a54790a5bf891c827e0ced => generated 0 bytes in 221 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 16:42:15.446550 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67c47b8b-a680-4ec2-afcb-89081c35010b None None] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:42:15.446874 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67c47b8b-a680-4ec2-afcb-89081c35010b None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:42:15.447188 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67c47b8b-a680-4ec2-afcb-89081c35010b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:42:15.468888 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.auth.core [None req-67c47b8b-a680-4ec2-afcb-89081c35010b None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81437) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:42:15.470148 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-67c47b8b-a680-4ec2-afcb-89081c35010b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:42:15.478823 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:42:15.549565 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1356/2702] 10.222.0.50 () {58 vars in 958 bytes} [Tue Sep 13 16:42:15 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:42:15.562654 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-120dbc1b-b810-49b6-a0ae-8831021d8e02 admin admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:42:15.564818 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-120dbc1b-b810-49b6-a0ae-8831021d8e02 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:42:15.625295 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-120dbc1b-b810-49b6-a0ae-8831021d8e02 admin admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:42:15.629819 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73777b6b-48ec-4c7e-bc08-f41bcc828cc6 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f436f07939d4855a21b195bc7566244', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:42:15.630432 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73777b6b-48ec-4c7e-bc08-f41bcc828cc6 service neutron] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:42:15.630710 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73777b6b-48ec-4c7e-bc08-f41bcc828cc6 service neutron] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:42:15.631012 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73777b6b-48ec-4c7e-bc08-f41bcc828cc6 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:42:15.640304 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73777b6b-48ec-4c7e-bc08-f41bcc828cc6 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:42:15.641775 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73777b6b-48ec-4c7e-bc08-f41bcc828cc6 service neutron] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:42:15.651676 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1347/2703] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 16:42:15 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 90 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 16:42:15.963714 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67c47b8b-a680-4ec2-afcb-89081c35010b None None] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:42:15.965430 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-67c47b8b-a680-4ec2-afcb-89081c35010b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:42:15.979118 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67c47b8b-a680-4ec2-afcb-89081c35010b None None] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:42:15.983936 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26889a5e-b649-44f8-bfa8-5d5893aa9f8f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:42:15.985501 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26889a5e-b649-44f8-bfa8-5d5893aa9f8f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:42:15.986239 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26889a5e-b649-44f8-bfa8-5d5893aa9f8f admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:42:15.986239 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26889a5e-b649-44f8-bfa8-5d5893aa9f8f admin admin] PATH_INFO: `/v3/projects/18456647e1c04144864ccd3ee60ea89a` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:42:15.987555 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26889a5e-b649-44f8-bfa8-5d5893aa9f8f admin admin] RBAC: Authorizing `identity:delete_project(project_id=18456647e1c04144864ccd3ee60ea89a)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:42:15.989486 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26889a5e-b649-44f8-bfa8-5d5893aa9f8f admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:42:16.040401 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-26889a5e-b649-44f8-bfa8-5d5893aa9f8f admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '18456647e1c04144864ccd3ee60ea89a', 'request_id': 'req-26889a5e-b649-44f8-bfa8-5d5893aa9f8f'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:42:16.050201 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-26889a5e-b649-44f8-bfa8-5d5893aa9f8f admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '18456647e1c04144864ccd3ee60ea89a', 'request_id': 'req-26889a5e-b649-44f8-bfa8-5d5893aa9f8f'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:42:16.057191 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-26889a5e-b649-44f8-bfa8-5d5893aa9f8f admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '18456647e1c04144864ccd3ee60ea89a', 'request_id': 'req-26889a5e-b649-44f8-bfa8-5d5893aa9f8f'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:42:16.069104 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-26889a5e-b649-44f8-bfa8-5d5893aa9f8f admin admin] The token cache is being invalidate because project 18456647e1c04144864ccd3ee60ea89a was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81437) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 13 16:42:16.069394 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-26889a5e-b649-44f8-bfa8-5d5893aa9f8f admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:42:16.070052 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-26889a5e-b649-44f8-bfa8-5d5893aa9f8f admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:42:16.071567 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1357/2704] 10.222.0.50 () {58 vars in 1233 bytes} [Tue Sep 13 16:42:15 2022] DELETE /identity/v3/projects/18456647e1c04144864ccd3ee60ea89a => generated 0 bytes in 108 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 16:42:16.075519 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73777b6b-48ec-4c7e-bc08-f41bcc828cc6 service neutron] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:42:16.085800 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73777b6b-48ec-4c7e-bc08-f41bcc828cc6 service neutron] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:42:16.148581 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26889a5e-b649-44f8-bfa8-5d5893aa9f8f admin admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:42:16.151348 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-26889a5e-b649-44f8-bfa8-5d5893aa9f8f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:42:16.156217 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f01573eb-6b66-4c09-846f-ac5bf2d50dfe admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:42:16.157678 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f01573eb-6b66-4c09-846f-ac5bf2d50dfe admin admin] REQUEST_METHOD: `DELETE` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:42:16.158424 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f01573eb-6b66-4c09-846f-ac5bf2d50dfe admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:42:16.158701 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f01573eb-6b66-4c09-846f-ac5bf2d50dfe admin admin] PATH_INFO: `/v3/users/3c3055d03a794d039b9a7f9f7ea5b4c4` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:42:16.159894 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f01573eb-6b66-4c09-846f-ac5bf2d50dfe admin admin] RBAC: Authorizing `identity:delete_user(user_id=3c3055d03a794d039b9a7f9f7ea5b4c4)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:42:16.161803 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f01573eb-6b66-4c09-846f-ac5bf2d50dfe admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:42:16.177080 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=88,keystone:DELETE=11,keystone:INSERT=1 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:42:16.223446 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-f01573eb-6b66-4c09-846f-ac5bf2d50dfe admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '3c3055d03a794d039b9a7f9f7ea5b4c4'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:42:16.241832 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26889a5e-b649-44f8-bfa8-5d5893aa9f8f admin admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:42:16.246836 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-f01573eb-6b66-4c09-846f-ac5bf2d50dfe admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '3c3055d03a794d039b9a7f9f7ea5b4c4'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:42:16.247306 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adab53bb-dd9b-4521-b926-f1c1cce831e5 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '31480b61418340bd88583f034f67412e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:42:16.248100 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adab53bb-dd9b-4521-b926-f1c1cce831e5 service placement] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:42:16.248395 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adab53bb-dd9b-4521-b926-f1c1cce831e5 service placement] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:42:16.248673 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adab53bb-dd9b-4521-b926-f1c1cce831e5 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:42:16.251089 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-adab53bb-dd9b-4521-b926-f1c1cce831e5 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:42:16.261443 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-f01573eb-6b66-4c09-846f-ac5bf2d50dfe admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '3c3055d03a794d039b9a7f9f7ea5b4c4'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:42:16.262131 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-f01573eb-6b66-4c09-846f-ac5bf2d50dfe admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '3c3055d03a794d039b9a7f9f7ea5b4c4'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:42:16.263483 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-adab53bb-dd9b-4521-b926-f1c1cce831e5 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:42:16.265069 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-adab53bb-dd9b-4521-b926-f1c1cce831e5 service placement] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:42:16.271330 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-f01573eb-6b66-4c09-846f-ac5bf2d50dfe admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '3c3055d03a794d039b9a7f9f7ea5b4c4'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:42:16.277190 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1358/2705] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 16:42:16 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 129 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 13 16:42:16.277480 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1348/2706] 10.222.0.50 () {58 vars in 1224 bytes} [Tue Sep 13 16:42:16 2022] DELETE /identity/v3/users/3c3055d03a794d039b9a7f9f7ea5b4c4 => generated 0 bytes in 202 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 16:42:16.589926 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adab53bb-dd9b-4521-b926-f1c1cce831e5 service placement] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:42:16.591757 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-adab53bb-dd9b-4521-b926-f1c1cce831e5 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:42:16.612870 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adab53bb-dd9b-4521-b926-f1c1cce831e5 service placement] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:42:16.642734 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:42:16.680163 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db1643ec-7b7b-4208-b9bd-fcfa82e88dcc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:42:16.680705 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db1643ec-7b7b-4208-b9bd-fcfa82e88dcc admin admin] REQUEST_METHOD: `DELETE` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:42:16.680980 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db1643ec-7b7b-4208-b9bd-fcfa82e88dcc admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:42:16.681255 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db1643ec-7b7b-4208-b9bd-fcfa82e88dcc admin admin] PATH_INFO: `/v3/projects/9a74c97db23a45b7bec0def4fa75a119` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:42:16.682493 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db1643ec-7b7b-4208-b9bd-fcfa82e88dcc admin admin] RBAC: Authorizing `identity:delete_project(project_id=9a74c97db23a45b7bec0def4fa75a119)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:42:16.684609 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db1643ec-7b7b-4208-b9bd-fcfa82e88dcc admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:42:16.736413 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-db1643ec-7b7b-4208-b9bd-fcfa82e88dcc admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '9a74c97db23a45b7bec0def4fa75a119', 'request_id': 'req-db1643ec-7b7b-4208-b9bd-fcfa82e88dcc'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:42:16.743850 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-db1643ec-7b7b-4208-b9bd-fcfa82e88dcc admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '9a74c97db23a45b7bec0def4fa75a119', 'request_id': 'req-db1643ec-7b7b-4208-b9bd-fcfa82e88dcc'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:42:16.750350 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-db1643ec-7b7b-4208-b9bd-fcfa82e88dcc admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '9a74c97db23a45b7bec0def4fa75a119', 'request_id': 'req-db1643ec-7b7b-4208-b9bd-fcfa82e88dcc'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:42:16.760800 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-db1643ec-7b7b-4208-b9bd-fcfa82e88dcc admin admin] The token cache is being invalidate because project 9a74c97db23a45b7bec0def4fa75a119 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81437) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 13 16:42:16.761091 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-db1643ec-7b7b-4208-b9bd-fcfa82e88dcc admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:42:16.761729 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-db1643ec-7b7b-4208-b9bd-fcfa82e88dcc admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:42:16.763316 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1359/2707] 10.222.0.50 () {58 vars in 1233 bytes} [Tue Sep 13 16:42:16 2022] DELETE /identity/v3/projects/9a74c97db23a45b7bec0def4fa75a119 => generated 0 bytes in 174 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 16:42:16.767353 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f01573eb-6b66-4c09-846f-ac5bf2d50dfe admin admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:42:16.768769 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-f01573eb-6b66-4c09-846f-ac5bf2d50dfe admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:42:16.781373 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f01573eb-6b66-4c09-846f-ac5bf2d50dfe admin admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:42:16.837827 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc75b4e5-fe32-4d9c-bc62-9b5dc3e0eba6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:42:16.838463 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc75b4e5-fe32-4d9c-bc62-9b5dc3e0eba6 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:42:16.838743 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc75b4e5-fe32-4d9c-bc62-9b5dc3e0eba6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:42:16.839060 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc75b4e5-fe32-4d9c-bc62-9b5dc3e0eba6 admin admin] PATH_INFO: `/v3/users/f1e337e487c445658091806a16cdbdfb` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:42:16.840176 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc75b4e5-fe32-4d9c-bc62-9b5dc3e0eba6 admin admin] RBAC: Authorizing `identity:delete_user(user_id=f1e337e487c445658091806a16cdbdfb)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:42:16.842106 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc75b4e5-fe32-4d9c-bc62-9b5dc3e0eba6 admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:42:16.888693 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-bc75b4e5-fe32-4d9c-bc62-9b5dc3e0eba6 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'f1e337e487c445658091806a16cdbdfb'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:42:16.896074 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-bc75b4e5-fe32-4d9c-bc62-9b5dc3e0eba6 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'f1e337e487c445658091806a16cdbdfb'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:42:16.908482 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-bc75b4e5-fe32-4d9c-bc62-9b5dc3e0eba6 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'f1e337e487c445658091806a16cdbdfb'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:42:16.909120 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-bc75b4e5-fe32-4d9c-bc62-9b5dc3e0eba6 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'f1e337e487c445658091806a16cdbdfb'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:42:16.915968 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-bc75b4e5-fe32-4d9c-bc62-9b5dc3e0eba6 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'f1e337e487c445658091806a16cdbdfb'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:42:16.921799 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1349/2708] 10.222.0.50 () {58 vars in 1224 bytes} [Tue Sep 13 16:42:16 2022] DELETE /identity/v3/users/f1e337e487c445658091806a16cdbdfb => generated 0 bytes in 155 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 16:42:17.200004 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db1643ec-7b7b-4208-b9bd-fcfa82e88dcc admin admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:42:17.201472 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-db1643ec-7b7b-4208-b9bd-fcfa82e88dcc admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:42:17.215373 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db1643ec-7b7b-4208-b9bd-fcfa82e88dcc admin admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:42:17.276818 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58750811-e958-44da-af55-202c9dd17530 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:42:17.277467 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58750811-e958-44da-af55-202c9dd17530 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:42:17.277784 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58750811-e958-44da-af55-202c9dd17530 admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:42:17.278065 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58750811-e958-44da-af55-202c9dd17530 admin admin] PATH_INFO: `/v3/projects/c079b4d3f9fe4f15a389d81cb2d5e469` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:42:17.279254 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58750811-e958-44da-af55-202c9dd17530 admin admin] RBAC: Authorizing `identity:delete_project(project_id=c079b4d3f9fe4f15a389d81cb2d5e469)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:42:17.281267 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58750811-e958-44da-af55-202c9dd17530 admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:42:17.331747 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-58750811-e958-44da-af55-202c9dd17530 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'c079b4d3f9fe4f15a389d81cb2d5e469', 'request_id': 'req-58750811-e958-44da-af55-202c9dd17530'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:42:17.339460 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-58750811-e958-44da-af55-202c9dd17530 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'c079b4d3f9fe4f15a389d81cb2d5e469', 'request_id': 'req-58750811-e958-44da-af55-202c9dd17530'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:42:17.341598 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=86,keystone:DELETE=12,keystone:INSERT=2 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:42:17.347772 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-58750811-e958-44da-af55-202c9dd17530 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'c079b4d3f9fe4f15a389d81cb2d5e469', 'request_id': 'req-58750811-e958-44da-af55-202c9dd17530'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:42:17.359244 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-58750811-e958-44da-af55-202c9dd17530 admin admin] The token cache is being invalidate because project c079b4d3f9fe4f15a389d81cb2d5e469 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81437) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 13 16:42:17.359542 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-58750811-e958-44da-af55-202c9dd17530 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:42:17.360359 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-58750811-e958-44da-af55-202c9dd17530 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:42:17.362049 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1360/2709] 10.222.0.50 () {58 vars in 1233 bytes} [Tue Sep 13 16:42:17 2022] DELETE /identity/v3/projects/c079b4d3f9fe4f15a389d81cb2d5e469 => generated 0 bytes in 162 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 16:42:17.366494 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a67e7ca2-146b-4d1b-85e2-20d2eb4e96e6 None None] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:42:17.366892 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a67e7ca2-146b-4d1b-85e2-20d2eb4e96e6 None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:42:17.367083 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a67e7ca2-146b-4d1b-85e2-20d2eb4e96e6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:42:17.387256 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.auth.core [None req-a67e7ca2-146b-4d1b-85e2-20d2eb4e96e6 None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81438) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:42:17.388512 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-a67e7ca2-146b-4d1b-85e2-20d2eb4e96e6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:42:17.393965 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=79,keystone:DELETE=19,keystone:INSERT=2 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:42:17.450908 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1350/2710] 10.222.0.50 () {58 vars in 958 bytes} [Tue Sep 13 16:42:17 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:42:17.460330 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18ae571c-0bc4-4db3-880b-d8e96e950909 None None] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:42:17.460616 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18ae571c-0bc4-4db3-880b-d8e96e950909 None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:42:17.460918 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18ae571c-0bc4-4db3-880b-d8e96e950909 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:42:17.481418 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.auth.core [None req-18ae571c-0bc4-4db3-880b-d8e96e950909 None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81437) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:42:17.482636 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-18ae571c-0bc4-4db3-880b-d8e96e950909 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:42:17.490684 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1361/2711] 10.222.0.50 () {58 vars in 958 bytes} [Tue Sep 13 16:42:17 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:42:17.494800 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a67e7ca2-146b-4d1b-85e2-20d2eb4e96e6 None None] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:42:17.505570 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a67e7ca2-146b-4d1b-85e2-20d2eb4e96e6 None None] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:42:17.510272 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e77d759-4da3-4a5d-afc3-2ea452c5c78e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:42:17.510974 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e77d759-4da3-4a5d-afc3-2ea452c5c78e admin admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:42:17.511202 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e77d759-4da3-4a5d-afc3-2ea452c5c78e admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:42:17.511472 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e77d759-4da3-4a5d-afc3-2ea452c5c78e admin admin] PATH_INFO: `/v3/domains` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:42:17.518053 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e77d759-4da3-4a5d-afc3-2ea452c5c78e admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81438) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 16:42:17.518053 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e77d759-4da3-4a5d-afc3-2ea452c5c78e admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:42:17.518053 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e77d759-4da3-4a5d-afc3-2ea452c5c78e admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:42:17.526379 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1351/2712] 10.222.0.50 () {58 vars in 1153 bytes} [Tue Sep 13 16:42:17 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 16:42:17.530667 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18ae571c-0bc4-4db3-880b-d8e96e950909 None None] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:42:17.540110 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18ae571c-0bc4-4db3-880b-d8e96e950909 None None] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:42:17.544645 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f070d7c9-08bc-44c6-9233-cdde2ddab227 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:42:17.545231 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f070d7c9-08bc-44c6-9233-cdde2ddab227 admin admin] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:42:17.545503 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f070d7c9-08bc-44c6-9233-cdde2ddab227 admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:42:17.545771 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f070d7c9-08bc-44c6-9233-cdde2ddab227 admin admin] PATH_INFO: `/v3/projects` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:42:17.546396 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f070d7c9-08bc-44c6-9233-cdde2ddab227 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:42:17.548713 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f070d7c9-08bc-44c6-9233-cdde2ddab227 admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:42:17.576326 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1362/2713] 10.222.0.50 () {60 vars in 1153 bytes} [Tue Sep 13 16:42:17 2022] POST /identity/v3/projects => generated 373 bytes in 46 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 16:42:17.581453 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e77d759-4da3-4a5d-afc3-2ea452c5c78e admin admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:42:17.591392 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e77d759-4da3-4a5d-afc3-2ea452c5c78e admin admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:42:17.649752 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7326251-f531-4468-a25e-4855a87a3e1b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:42:17.650359 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7326251-f531-4468-a25e-4855a87a3e1b admin admin] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:42:17.650637 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7326251-f531-4468-a25e-4855a87a3e1b admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:42:17.650919 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7326251-f531-4468-a25e-4855a87a3e1b admin admin] PATH_INFO: `/v3/users` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:42:17.651584 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7326251-f531-4468-a25e-4855a87a3e1b admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:42:17.653502 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7326251-f531-4468-a25e-4855a87a3e1b admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:42:17.674293 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1352/2714] 10.222.0.50 () {60 vars in 1144 bytes} [Tue Sep 13 16:42:17 2022] POST /identity/v3/users => generated 287 bytes in 93 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 16:42:17.678499 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f070d7c9-08bc-44c6-9233-cdde2ddab227 admin admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:42:17.689328 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f070d7c9-08bc-44c6-9233-cdde2ddab227 admin admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:42:17.693855 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57343b4c-e5eb-49fd-9f59-12d3456a4d43 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:42:17.694447 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57343b4c-e5eb-49fd-9f59-12d3456a4d43 admin admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:42:17.694721 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57343b4c-e5eb-49fd-9f59-12d3456a4d43 admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:42:17.695022 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57343b4c-e5eb-49fd-9f59-12d3456a4d43 admin admin] PATH_INFO: `/v3/roles` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:42:17.695656 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57343b4c-e5eb-49fd-9f59-12d3456a4d43 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:42:17.697238 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57343b4c-e5eb-49fd-9f59-12d3456a4d43 admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:42:17.708508 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1363/2715] 10.222.0.50 () {58 vars in 1122 bytes} [Tue Sep 13 16:42:17 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 16:42:17.712316 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7326251-f531-4468-a25e-4855a87a3e1b admin admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:42:17.722227 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7326251-f531-4468-a25e-4855a87a3e1b admin admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:42:17.726851 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6ce3d1f-bf56-47fc-9afd-5f74acabc39b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:42:17.727478 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6ce3d1f-bf56-47fc-9afd-5f74acabc39b admin admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:42:17.727760 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6ce3d1f-bf56-47fc-9afd-5f74acabc39b admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:42:17.728037 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6ce3d1f-bf56-47fc-9afd-5f74acabc39b admin admin] PATH_INFO: `/v3/roles` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:42:17.728669 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6ce3d1f-bf56-47fc-9afd-5f74acabc39b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:42:17.730238 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6ce3d1f-bf56-47fc-9afd-5f74acabc39b admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:42:17.741235 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1353/2716] 10.222.0.50 () {58 vars in 1122 bytes} [Tue Sep 13 16:42:17 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 16:42:17.745211 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57343b4c-e5eb-49fd-9f59-12d3456a4d43 admin admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:42:17.755783 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57343b4c-e5eb-49fd-9f59-12d3456a4d43 admin admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:42:17.760424 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84ccae88-b728-43fc-abbb-cbc7d42a41a7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:42:17.761085 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84ccae88-b728-43fc-abbb-cbc7d42a41a7 admin admin] REQUEST_METHOD: `PUT` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:42:17.761367 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84ccae88-b728-43fc-abbb-cbc7d42a41a7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:42:17.761643 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84ccae88-b728-43fc-abbb-cbc7d42a41a7 admin admin] PATH_INFO: `/v3/projects/b70b7595c0e44310ba172e35bc6e8fe3/users/94f08e22c26f4c5d8a6789b1d7a0e1a7/roles/15db4aaca3264ae29bde38903ba43471` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:42:17.776802 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84ccae88-b728-43fc-abbb-cbc7d42a41a7 admin admin] RBAC: Authorizing `identity:create_grant(project_id=b70b7595c0e44310ba172e35bc6e8fe3, user_id=94f08e22c26f4c5d8a6789b1d7a0e1a7, role_id=15db4aaca3264ae29bde38903ba43471)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:42:17.779521 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84ccae88-b728-43fc-abbb-cbc7d42a41a7 admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:42:17.793422 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1364/2717] 10.222.0.50 () {60 vars in 1483 bytes} [Tue Sep 13 16:42:17 2022] PUT /identity/v3/projects/b70b7595c0e44310ba172e35bc6e8fe3/users/94f08e22c26f4c5d8a6789b1d7a0e1a7/roles/15db4aaca3264ae29bde38903ba43471 => generated 0 bytes in 49 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 13 16:42:17.799754 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-142da571-c817-4b50-9048-30fe8a4a9d79 None None] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:42:17.800262 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-142da571-c817-4b50-9048-30fe8a4a9d79 None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:42:17.800762 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-142da571-c817-4b50-9048-30fe8a4a9d79 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:42:17.820392 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.auth.core [None req-142da571-c817-4b50-9048-30fe8a4a9d79 None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81438) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:42:17.821601 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-142da571-c817-4b50-9048-30fe8a4a9d79 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:42:17.885698 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1354/2718] 10.222.0.50 () {58 vars in 958 bytes} [Tue Sep 13 16:42:17 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:42:17.895244 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84ccae88-b728-43fc-abbb-cbc7d42a41a7 admin admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:42:17.896922 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-84ccae88-b728-43fc-abbb-cbc7d42a41a7 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:42:17.955451 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84ccae88-b728-43fc-abbb-cbc7d42a41a7 admin admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:42:17.959146 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a739e1f7-27bb-44c2-9b20-9ce274b9038d service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f436f07939d4855a21b195bc7566244', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:42:17.959749 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a739e1f7-27bb-44c2-9b20-9ce274b9038d service neutron] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:42:17.960026 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a739e1f7-27bb-44c2-9b20-9ce274b9038d service neutron] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:42:17.960298 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a739e1f7-27bb-44c2-9b20-9ce274b9038d service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:42:17.969056 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a739e1f7-27bb-44c2-9b20-9ce274b9038d service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:42:17.970490 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a739e1f7-27bb-44c2-9b20-9ce274b9038d service neutron] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:42:17.979766 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1365/2719] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 16:42:17 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 85 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 16:42:18.652928 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae741c8e-74e8-4742-bd82-f59806e8ef2f None None] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:42:18.654310 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae741c8e-74e8-4742-bd82-f59806e8ef2f None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:42:18.654791 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae741c8e-74e8-4742-bd82-f59806e8ef2f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:42:18.707489 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.auth.core [None req-ae741c8e-74e8-4742-bd82-f59806e8ef2f None None] MFA Rules not processed for user `94f08e22c26f4c5d8a6789b1d7a0e1a7`. Rule list: `[]` (Enabled: `True`). {{(pid=81438) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:42:18.709056 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-ae741c8e-74e8-4742-bd82-f59806e8ef2f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:42:18.715351 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:42:18.808111 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1355/2720] 10.222.0.50 () {58 vars in 958 bytes} [Tue Sep 13 16:42:18 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 157 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:42:18.819267 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a739e1f7-27bb-44c2-9b20-9ce274b9038d service neutron] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:42:18.820751 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-a739e1f7-27bb-44c2-9b20-9ce274b9038d service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:42:18.881174 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a739e1f7-27bb-44c2-9b20-9ce274b9038d service neutron] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:42:18.885861 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0f0ee2a-0213-4308-a7ea-10adb670ca86 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7fd6b5afd67b48f9b0f0597ccd4cf4df', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:42:18.886564 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0f0ee2a-0213-4308-a7ea-10adb670ca86 service cinder] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:42:18.886840 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0f0ee2a-0213-4308-a7ea-10adb670ca86 service cinder] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:42:18.887134 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0f0ee2a-0213-4308-a7ea-10adb670ca86 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:42:18.896448 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0f0ee2a-0213-4308-a7ea-10adb670ca86 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:42:18.897875 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0f0ee2a-0213-4308-a7ea-10adb670ca86 service cinder] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:42:18.906856 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1366/2721] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 16:42:18 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 88 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 16:42:25.544303 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-171389c9-10a8-4aef-99fd-d836d8c8617d None None] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:42:25.544845 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-171389c9-10a8-4aef-99fd-d836d8c8617d None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:42:25.544845 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-171389c9-10a8-4aef-99fd-d836d8c8617d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:42:25.566860 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.auth.core [None req-171389c9-10a8-4aef-99fd-d836d8c8617d None None] MFA Rules not processed for user `94f08e22c26f4c5d8a6789b1d7a0e1a7`. Rule list: `[]` (Enabled: `True`). {{(pid=81438) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:42:25.568085 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-171389c9-10a8-4aef-99fd-d836d8c8617d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:42:25.574888 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1356/2722] 10.222.0.50 () {58 vars in 958 bytes} [Tue Sep 13 16:42:25 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:42:25.816974 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0f0ee2a-0213-4308-a7ea-10adb670ca86 service cinder] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:42:25.826713 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0f0ee2a-0213-4308-a7ea-10adb670ca86 service cinder] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:42:25.830646 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b88bcf6-d8eb-4226-be0a-44221587affa service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f436f07939d4855a21b195bc7566244', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:42:25.831341 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b88bcf6-d8eb-4226-be0a-44221587affa service neutron] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:42:25.831723 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b88bcf6-d8eb-4226-be0a-44221587affa service neutron] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:42:25.832061 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b88bcf6-d8eb-4226-be0a-44221587affa service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:42:25.833862 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-2b88bcf6-d8eb-4226-be0a-44221587affa service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:42:25.844946 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b88bcf6-d8eb-4226-be0a-44221587affa service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:42:25.846419 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b88bcf6-d8eb-4226-be0a-44221587affa service neutron] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:42:25.860677 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1367/2723] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 16:42:25 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 38 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 16:42:26.053032 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-171389c9-10a8-4aef-99fd-d836d8c8617d None None] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:42:26.054519 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-171389c9-10a8-4aef-99fd-d836d8c8617d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:42:26.068195 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-171389c9-10a8-4aef-99fd-d836d8c8617d None None] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:42:26.072728 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-187d5a5b-7199-436a-9ec5-af515eb6759e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:42:26.073365 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-187d5a5b-7199-436a-9ec5-af515eb6759e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:42:26.073648 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-187d5a5b-7199-436a-9ec5-af515eb6759e admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:42:26.073928 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-187d5a5b-7199-436a-9ec5-af515eb6759e admin admin] PATH_INFO: `/v3/users/3c322eaf6cee42b0a24faf6fb61d227d` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:42:26.075087 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-187d5a5b-7199-436a-9ec5-af515eb6759e admin admin] RBAC: Authorizing `identity:delete_user(user_id=3c322eaf6cee42b0a24faf6fb61d227d)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:42:26.076984 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-187d5a5b-7199-436a-9ec5-af515eb6759e admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:42:26.128075 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-187d5a5b-7199-436a-9ec5-af515eb6759e admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '3c322eaf6cee42b0a24faf6fb61d227d'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:42:26.135240 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-187d5a5b-7199-436a-9ec5-af515eb6759e admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '3c322eaf6cee42b0a24faf6fb61d227d'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:42:26.152836 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-187d5a5b-7199-436a-9ec5-af515eb6759e admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '3c322eaf6cee42b0a24faf6fb61d227d'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:42:26.153741 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-187d5a5b-7199-436a-9ec5-af515eb6759e admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '3c322eaf6cee42b0a24faf6fb61d227d'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:42:26.163156 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-187d5a5b-7199-436a-9ec5-af515eb6759e admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '3c322eaf6cee42b0a24faf6fb61d227d'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:42:26.170120 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1357/2724] 10.222.0.50 () {58 vars in 1224 bytes} [Tue Sep 13 16:42:26 2022] DELETE /identity/v3/users/3c322eaf6cee42b0a24faf6fb61d227d => generated 0 bytes in 118 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 16:42:26.174715 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-138ebf96-2c81-4eab-89e5-50d78ee43569 None None] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:42:26.174919 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-138ebf96-2c81-4eab-89e5-50d78ee43569 None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:42:26.175211 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-138ebf96-2c81-4eab-89e5-50d78ee43569 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:42:26.194402 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.auth.core [None req-138ebf96-2c81-4eab-89e5-50d78ee43569 None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81437) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:42:26.195835 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-138ebf96-2c81-4eab-89e5-50d78ee43569 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:42:26.206835 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=96,keystone:DELETE=2,keystone:INSERT=2 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:42:26.283048 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1368/2725] 10.222.0.50 () {58 vars in 958 bytes} [Tue Sep 13 16:42:26 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:42:26.290220 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-187d5a5b-7199-436a-9ec5-af515eb6759e admin admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:42:26.291935 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-187d5a5b-7199-436a-9ec5-af515eb6759e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:42:26.362704 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-187d5a5b-7199-436a-9ec5-af515eb6759e admin admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:42:26.366415 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e520079-024d-4e58-9590-5657e9056da0 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f436f07939d4855a21b195bc7566244', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:42:26.367089 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e520079-024d-4e58-9590-5657e9056da0 service neutron] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:42:26.367376 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e520079-024d-4e58-9590-5657e9056da0 service neutron] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:42:26.367655 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e520079-024d-4e58-9590-5657e9056da0 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:42:26.379497 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e520079-024d-4e58-9590-5657e9056da0 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:42:26.381153 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e520079-024d-4e58-9590-5657e9056da0 service neutron] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:42:26.397966 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1358/2726] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 16:42:26 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 108 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 13 16:42:26.809087 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-138ebf96-2c81-4eab-89e5-50d78ee43569 None None] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:42:26.810991 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-138ebf96-2c81-4eab-89e5-50d78ee43569 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:42:26.823745 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-138ebf96-2c81-4eab-89e5-50d78ee43569 None None] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:42:26.830804 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ebec3b2-15eb-46a6-967a-27f3a10e9b50 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:42:26.831386 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ebec3b2-15eb-46a6-967a-27f3a10e9b50 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:42:26.838214 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ebec3b2-15eb-46a6-967a-27f3a10e9b50 admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:42:26.838650 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ebec3b2-15eb-46a6-967a-27f3a10e9b50 admin admin] PATH_INFO: `/v3/projects/414f0a9cc70f45ec9c9dfc55964392c0` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:42:26.840019 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ebec3b2-15eb-46a6-967a-27f3a10e9b50 admin admin] RBAC: Authorizing `identity:delete_project(project_id=414f0a9cc70f45ec9c9dfc55964392c0)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:42:26.841944 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ebec3b2-15eb-46a6-967a-27f3a10e9b50 admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:42:26.895883 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-1ebec3b2-15eb-46a6-967a-27f3a10e9b50 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '414f0a9cc70f45ec9c9dfc55964392c0', 'request_id': 'req-1ebec3b2-15eb-46a6-967a-27f3a10e9b50'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:42:26.907229 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-1ebec3b2-15eb-46a6-967a-27f3a10e9b50 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '414f0a9cc70f45ec9c9dfc55964392c0', 'request_id': 'req-1ebec3b2-15eb-46a6-967a-27f3a10e9b50'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:42:26.916267 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-1ebec3b2-15eb-46a6-967a-27f3a10e9b50 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '414f0a9cc70f45ec9c9dfc55964392c0', 'request_id': 'req-1ebec3b2-15eb-46a6-967a-27f3a10e9b50'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:42:26.930126 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-1ebec3b2-15eb-46a6-967a-27f3a10e9b50 admin admin] The token cache is being invalidate because project 414f0a9cc70f45ec9c9dfc55964392c0 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81437) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 13 16:42:26.930609 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-1ebec3b2-15eb-46a6-967a-27f3a10e9b50 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:42:26.931344 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-1ebec3b2-15eb-46a6-967a-27f3a10e9b50 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:42:26.942932 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1369/2727] 10.222.0.50 () {58 vars in 1233 bytes} [Tue Sep 13 16:42:26 2022] DELETE /identity/v3/projects/414f0a9cc70f45ec9c9dfc55964392c0 => generated 0 bytes in 125 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 16:42:34.927798 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e520079-024d-4e58-9590-5657e9056da0 service neutron] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:42:34.929200 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-9e520079-024d-4e58-9590-5657e9056da0 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:42:34.942168 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e520079-024d-4e58-9590-5657e9056da0 service neutron] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:42:34.951647 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:42:35.004450 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e39caaa-e74c-4bb2-9f41-32ac22367295 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:42:35.005082 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e39caaa-e74c-4bb2-9f41-32ac22367295 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:42:35.005358 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e39caaa-e74c-4bb2-9f41-32ac22367295 admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:42:35.005632 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e39caaa-e74c-4bb2-9f41-32ac22367295 admin admin] PATH_INFO: `/v3/users/79b893555a1849e7901df3a2061ac370` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:42:35.006763 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e39caaa-e74c-4bb2-9f41-32ac22367295 admin admin] RBAC: Authorizing `identity:delete_user(user_id=79b893555a1849e7901df3a2061ac370)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:42:35.008804 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e39caaa-e74c-4bb2-9f41-32ac22367295 admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:42:35.066201 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-5e39caaa-e74c-4bb2-9f41-32ac22367295 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '79b893555a1849e7901df3a2061ac370'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:42:35.073874 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-5e39caaa-e74c-4bb2-9f41-32ac22367295 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '79b893555a1849e7901df3a2061ac370'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:42:35.086748 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-5e39caaa-e74c-4bb2-9f41-32ac22367295 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '79b893555a1849e7901df3a2061ac370'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:42:35.087476 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-5e39caaa-e74c-4bb2-9f41-32ac22367295 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '79b893555a1849e7901df3a2061ac370'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:42:35.094229 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-5e39caaa-e74c-4bb2-9f41-32ac22367295 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '79b893555a1849e7901df3a2061ac370'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:42:35.100280 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1359/2728] 10.222.0.50 () {58 vars in 1224 bytes} [Tue Sep 13 16:42:34 2022] DELETE /identity/v3/users/79b893555a1849e7901df3a2061ac370 => generated 0 bytes in 173 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 16:42:35.104605 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16e01bb5-f305-4b3f-87c2-d0245301bb96 None None] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:42:35.105088 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16e01bb5-f305-4b3f-87c2-d0245301bb96 None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:42:35.105197 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16e01bb5-f305-4b3f-87c2-d0245301bb96 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:42:35.124925 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.auth.core [None req-16e01bb5-f305-4b3f-87c2-d0245301bb96 None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81437) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:42:35.126720 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-16e01bb5-f305-4b3f-87c2-d0245301bb96 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:42:35.185343 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1370/2729] 10.222.0.50 () {58 vars in 958 bytes} [Tue Sep 13 16:42:35 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 82 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:42:35.195682 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e39caaa-e74c-4bb2-9f41-32ac22367295 admin admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:42:35.197135 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-5e39caaa-e74c-4bb2-9f41-32ac22367295 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:42:35.253889 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e39caaa-e74c-4bb2-9f41-32ac22367295 admin admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:42:35.257679 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03242e93-ec15-415a-b2fb-dccad9a7e485 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f436f07939d4855a21b195bc7566244', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:42:35.258312 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03242e93-ec15-415a-b2fb-dccad9a7e485 service neutron] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:42:35.258577 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03242e93-ec15-415a-b2fb-dccad9a7e485 service neutron] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:42:35.258852 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03242e93-ec15-415a-b2fb-dccad9a7e485 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:42:35.267966 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03242e93-ec15-415a-b2fb-dccad9a7e485 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:42:35.269409 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03242e93-ec15-415a-b2fb-dccad9a7e485 service neutron] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:42:35.278714 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1360/2730] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 16:42:35 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 16:42:35.557382 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16e01bb5-f305-4b3f-87c2-d0245301bb96 None None] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:42:35.559073 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-16e01bb5-f305-4b3f-87c2-d0245301bb96 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:42:35.575272 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16e01bb5-f305-4b3f-87c2-d0245301bb96 None None] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:42:35.582291 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14deb670-54cb-4412-b26e-5739a24b2075 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:42:35.583576 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14deb670-54cb-4412-b26e-5739a24b2075 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:42:35.584033 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14deb670-54cb-4412-b26e-5739a24b2075 admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:42:35.584318 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14deb670-54cb-4412-b26e-5739a24b2075 admin admin] PATH_INFO: `/v3/projects/395df427f71c403c8617c8aa5592b254` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:42:35.585440 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14deb670-54cb-4412-b26e-5739a24b2075 admin admin] RBAC: Authorizing `identity:delete_project(project_id=395df427f71c403c8617c8aa5592b254)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:42:35.587388 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14deb670-54cb-4412-b26e-5739a24b2075 admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:42:35.629853 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=90,keystone:DELETE=9,keystone:INSERT=1 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:42:35.643544 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-14deb670-54cb-4412-b26e-5739a24b2075 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '395df427f71c403c8617c8aa5592b254', 'request_id': 'req-14deb670-54cb-4412-b26e-5739a24b2075'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:42:35.651363 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-14deb670-54cb-4412-b26e-5739a24b2075 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '395df427f71c403c8617c8aa5592b254', 'request_id': 'req-14deb670-54cb-4412-b26e-5739a24b2075'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:42:35.658745 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-14deb670-54cb-4412-b26e-5739a24b2075 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '395df427f71c403c8617c8aa5592b254', 'request_id': 'req-14deb670-54cb-4412-b26e-5739a24b2075'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:42:35.670529 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-14deb670-54cb-4412-b26e-5739a24b2075 admin admin] The token cache is being invalidate because project 395df427f71c403c8617c8aa5592b254 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81437) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 13 16:42:35.670890 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-14deb670-54cb-4412-b26e-5739a24b2075 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:42:35.671524 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-14deb670-54cb-4412-b26e-5739a24b2075 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:42:35.673318 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1371/2731] 10.222.0.50 () {58 vars in 1233 bytes} [Tue Sep 13 16:42:35 2022] DELETE /identity/v3/projects/395df427f71c403c8617c8aa5592b254 => generated 0 bytes in 116 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 16:42:35.678403 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7ec3626-a9ee-4f4f-b030-4f735a5225a9 None None] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:42:35.678698 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7ec3626-a9ee-4f4f-b030-4f735a5225a9 None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:42:35.679019 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7ec3626-a9ee-4f4f-b030-4f735a5225a9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:42:35.701476 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.auth.core [None req-c7ec3626-a9ee-4f4f-b030-4f735a5225a9 None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81438) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:42:35.702721 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-c7ec3626-a9ee-4f4f-b030-4f735a5225a9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:42:35.736028 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:42:35.766992 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1361/2732] 10.222.0.50 () {58 vars in 958 bytes} [Tue Sep 13 16:42:35 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:42:35.775000 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c653f0a-12e6-422b-b687-d47e1cf02bfd None None] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:42:35.775295 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c653f0a-12e6-422b-b687-d47e1cf02bfd None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:42:35.775576 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c653f0a-12e6-422b-b687-d47e1cf02bfd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:42:35.796212 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.auth.core [None req-4c653f0a-12e6-422b-b687-d47e1cf02bfd None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81437) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:42:35.797579 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-4c653f0a-12e6-422b-b687-d47e1cf02bfd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:42:35.805951 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1372/2733] 10.222.0.50 () {58 vars in 958 bytes} [Tue Sep 13 16:42:35 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:42:35.809965 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7ec3626-a9ee-4f4f-b030-4f735a5225a9 None None] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:42:35.819849 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7ec3626-a9ee-4f4f-b030-4f735a5225a9 None None] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:42:35.824605 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b47374c2-d779-4e64-9604-44656525c408 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:42:35.825210 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b47374c2-d779-4e64-9604-44656525c408 admin admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:42:35.825489 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b47374c2-d779-4e64-9604-44656525c408 admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:42:35.825759 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b47374c2-d779-4e64-9604-44656525c408 admin admin] PATH_INFO: `/v3/domains` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:42:35.826403 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b47374c2-d779-4e64-9604-44656525c408 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81438) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 16:42:35.826702 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b47374c2-d779-4e64-9604-44656525c408 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:42:35.828909 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b47374c2-d779-4e64-9604-44656525c408 admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:42:35.841134 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1362/2734] 10.222.0.50 () {58 vars in 1153 bytes} [Tue Sep 13 16:42:35 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 16:42:35.846237 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c653f0a-12e6-422b-b687-d47e1cf02bfd None None] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:42:35.855607 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c653f0a-12e6-422b-b687-d47e1cf02bfd None None] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:42:35.861050 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f32fb5f6-b1d0-431b-bd6e-80877b41e497 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:42:35.861748 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f32fb5f6-b1d0-431b-bd6e-80877b41e497 admin admin] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:42:35.862033 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f32fb5f6-b1d0-431b-bd6e-80877b41e497 admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:42:35.862305 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f32fb5f6-b1d0-431b-bd6e-80877b41e497 admin admin] PATH_INFO: `/v3/projects` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:42:35.862982 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f32fb5f6-b1d0-431b-bd6e-80877b41e497 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:42:35.864893 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f32fb5f6-b1d0-431b-bd6e-80877b41e497 admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:42:35.899350 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1373/2735] 10.222.0.50 () {60 vars in 1153 bytes} [Tue Sep 13 16:42:35 2022] POST /identity/v3/projects => generated 369 bytes in 50 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 16:42:35.900764 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b47374c2-d779-4e64-9604-44656525c408 admin admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:42:35.910820 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b47374c2-d779-4e64-9604-44656525c408 admin admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:42:35.969369 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f32fb5f6-b1d0-431b-bd6e-80877b41e497 admin admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:42:35.971338 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-f32fb5f6-b1d0-431b-bd6e-80877b41e497 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:42:35.977611 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d920058-fff1-496f-aa2e-6ded7fb71870 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:42:35.978383 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d920058-fff1-496f-aa2e-6ded7fb71870 admin admin] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:42:35.978669 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d920058-fff1-496f-aa2e-6ded7fb71870 admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:42:35.979066 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d920058-fff1-496f-aa2e-6ded7fb71870 admin admin] PATH_INFO: `/v3/users` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:42:35.979857 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d920058-fff1-496f-aa2e-6ded7fb71870 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:42:35.981887 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d920058-fff1-496f-aa2e-6ded7fb71870 admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:42:36.010407 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1363/2736] 10.222.0.50 () {60 vars in 1144 bytes} [Tue Sep 13 16:42:35 2022] POST /identity/v3/users => generated 285 bytes in 107 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 16:42:36.011879 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d920058-fff1-496f-aa2e-6ded7fb71870 admin admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:42:36.023499 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d920058-fff1-496f-aa2e-6ded7fb71870 admin admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:42:36.028233 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8b79895-79a0-46a6-bbd5-8839420b2f16 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:42:36.028934 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8b79895-79a0-46a6-bbd5-8839420b2f16 admin admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:42:36.029124 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8b79895-79a0-46a6-bbd5-8839420b2f16 admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:42:36.029383 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8b79895-79a0-46a6-bbd5-8839420b2f16 admin admin] PATH_INFO: `/v3/roles` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:42:36.030348 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8b79895-79a0-46a6-bbd5-8839420b2f16 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:42:36.031886 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8b79895-79a0-46a6-bbd5-8839420b2f16 admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:42:36.050213 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1364/2737] 10.222.0.50 () {58 vars in 1122 bytes} [Tue Sep 13 16:42:36 2022] GET /identity/v3/roles => generated 1420 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 16:42:36.054286 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8b79895-79a0-46a6-bbd5-8839420b2f16 admin admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:42:36.065336 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8b79895-79a0-46a6-bbd5-8839420b2f16 admin admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:42:36.070511 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba79a74f-3905-46d0-b6e9-799d343ccd56 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:42:36.071174 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba79a74f-3905-46d0-b6e9-799d343ccd56 admin admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:42:36.071456 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba79a74f-3905-46d0-b6e9-799d343ccd56 admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:42:36.071728 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba79a74f-3905-46d0-b6e9-799d343ccd56 admin admin] PATH_INFO: `/v3/roles` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:42:36.072368 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba79a74f-3905-46d0-b6e9-799d343ccd56 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:42:36.073951 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba79a74f-3905-46d0-b6e9-799d343ccd56 admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:42:36.088505 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1365/2738] 10.222.0.50 () {58 vars in 1122 bytes} [Tue Sep 13 16:42:36 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 13 16:42:36.088646 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f32fb5f6-b1d0-431b-bd6e-80877b41e497 admin admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:42:36.089731 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba79a74f-3905-46d0-b6e9-799d343ccd56 admin admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:42:36.093938 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a1a7b41-f149-48a7-91b6-da7d9a3af01c service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4e039b46c5e3445bbb39904b00c5430e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'admin', 'ResellerAdmin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:42:36.094539 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a1a7b41-f149-48a7-91b6-da7d9a3af01c service nova] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:42:36.094819 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a1a7b41-f149-48a7-91b6-da7d9a3af01c service nova] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:42:36.095140 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a1a7b41-f149-48a7-91b6-da7d9a3af01c service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:42:36.097517 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-8a1a7b41-f149-48a7-91b6-da7d9a3af01c service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:42:36.101224 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba79a74f-3905-46d0-b6e9-799d343ccd56 admin admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:42:36.106398 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12f5174a-4bb1-42bc-a733-2d0e5b9ecb15 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:42:36.108342 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12f5174a-4bb1-42bc-a733-2d0e5b9ecb15 admin admin] REQUEST_METHOD: `PUT` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:42:36.108623 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12f5174a-4bb1-42bc-a733-2d0e5b9ecb15 admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:42:36.108901 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12f5174a-4bb1-42bc-a733-2d0e5b9ecb15 admin admin] PATH_INFO: `/v3/projects/8fafcd78e2fd4f5fb0eb12a8612ed1e0/users/aad181425a954b31ab1ccf2f9657e2f2/roles/15db4aaca3264ae29bde38903ba43471` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:42:36.110479 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a1a7b41-f149-48a7-91b6-da7d9a3af01c service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:42:36.112934 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a1a7b41-f149-48a7-91b6-da7d9a3af01c service nova] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:42:36.124335 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1374/2739] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 16:42:35 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 155 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 13 16:42:36.126860 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12f5174a-4bb1-42bc-a733-2d0e5b9ecb15 admin admin] RBAC: Authorizing `identity:create_grant(project_id=8fafcd78e2fd4f5fb0eb12a8612ed1e0, user_id=aad181425a954b31ab1ccf2f9657e2f2, role_id=15db4aaca3264ae29bde38903ba43471)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:42:36.129891 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12f5174a-4bb1-42bc-a733-2d0e5b9ecb15 admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:42:36.153541 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1366/2740] 10.222.0.50 () {60 vars in 1483 bytes} [Tue Sep 13 16:42:36 2022] PUT /identity/v3/projects/8fafcd78e2fd4f5fb0eb12a8612ed1e0/users/aad181425a954b31ab1ccf2f9657e2f2/roles/15db4aaca3264ae29bde38903ba43471 => generated 0 bytes in 55 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 16:42:36.154782 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da60a883-6fa1-48ba-81fb-1309a01739f6 None None] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:42:36.155093 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da60a883-6fa1-48ba-81fb-1309a01739f6 None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:42:36.155369 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da60a883-6fa1-48ba-81fb-1309a01739f6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:42:36.194348 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.auth.core [None req-da60a883-6fa1-48ba-81fb-1309a01739f6 None None] MFA Rules not processed for user `aad181425a954b31ab1ccf2f9657e2f2`. Rule list: `[]` (Enabled: `True`). {{(pid=81437) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:42:36.195839 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-da60a883-6fa1-48ba-81fb-1309a01739f6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:42:36.292601 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:DELETE=5,keystone:SELECT=93,keystone:INSERT=2 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:42:36.309678 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1375/2741] 10.222.0.50 () {58 vars in 958 bytes} [Tue Sep 13 16:42:36 2022] POST /identity/v3/auth/tokens => generated 3246 bytes in 156 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:42:36.321455 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12f5174a-4bb1-42bc-a733-2d0e5b9ecb15 admin admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:42:36.323188 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-12f5174a-4bb1-42bc-a733-2d0e5b9ecb15 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:42:36.414375 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:42:36.419734 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12f5174a-4bb1-42bc-a733-2d0e5b9ecb15 admin admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:42:36.425661 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd6f986d-1cba-4cdd-b50a-92b2d5e2680d service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6ab5f8018f464be69d4af7683f553738', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:42:36.426321 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd6f986d-1cba-4cdd-b50a-92b2d5e2680d service glance] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:42:36.426603 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd6f986d-1cba-4cdd-b50a-92b2d5e2680d service glance] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:42:36.426878 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd6f986d-1cba-4cdd-b50a-92b2d5e2680d service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:42:36.436676 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd6f986d-1cba-4cdd-b50a-92b2d5e2680d service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:42:36.438642 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd6f986d-1cba-4cdd-b50a-92b2d5e2680d service glance] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:42:36.448126 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1367/2742] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 16:42:36 2022] GET /identity/v3/auth/tokens => generated 3246 bytes in 127 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 16:42:39.582746 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da60a883-6fa1-48ba-81fb-1309a01739f6 None None] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:42:39.585580 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-da60a883-6fa1-48ba-81fb-1309a01739f6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:42:39.648407 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da60a883-6fa1-48ba-81fb-1309a01739f6 None None] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:42:39.651988 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32525f60-c96f-4306-a552-0aa4ec7b23fc service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7fd6b5afd67b48f9b0f0597ccd4cf4df', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:42:39.652659 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32525f60-c96f-4306-a552-0aa4ec7b23fc service cinder] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:42:39.652996 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32525f60-c96f-4306-a552-0aa4ec7b23fc service cinder] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:42:39.653271 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32525f60-c96f-4306-a552-0aa4ec7b23fc service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:42:39.655026 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-32525f60-c96f-4306-a552-0aa4ec7b23fc service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:42:39.752264 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32525f60-c96f-4306-a552-0aa4ec7b23fc service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:42:39.754702 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32525f60-c96f-4306-a552-0aa4ec7b23fc service cinder] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:42:39.786240 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1376/2743] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 16:42:39 2022] GET /identity/v3/auth/tokens => generated 3266 bytes in 204 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 16:42:40.132104 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd6f986d-1cba-4cdd-b50a-92b2d5e2680d service glance] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:42:40.133575 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-dd6f986d-1cba-4cdd-b50a-92b2d5e2680d service glance] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:42:40.196804 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd6f986d-1cba-4cdd-b50a-92b2d5e2680d service glance] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:42:40.200328 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4686d3c-27c9-481d-95f5-4b4afc719263 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f436f07939d4855a21b195bc7566244', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:42:40.200920 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4686d3c-27c9-481d-95f5-4b4afc719263 service neutron] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:42:40.201196 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4686d3c-27c9-481d-95f5-4b4afc719263 service neutron] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:42:40.201465 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4686d3c-27c9-481d-95f5-4b4afc719263 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:42:40.203189 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-c4686d3c-27c9-481d-95f5-4b4afc719263 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:42:40.272323 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4686d3c-27c9-481d-95f5-4b4afc719263 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:42:40.273736 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4686d3c-27c9-481d-95f5-4b4afc719263 service neutron] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:42:40.286529 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1368/2744] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 16:42:40 2022] GET /identity/v3/auth/tokens => generated 3323 bytes in 155 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 16:42:42.063709 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32525f60-c96f-4306-a552-0aa4ec7b23fc service cinder] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:42:42.219865 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32525f60-c96f-4306-a552-0aa4ec7b23fc service cinder] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:42:42.234159 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5eaf07c-89a4-48e4-a896-22018702f391 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4e039b46c5e3445bbb39904b00c5430e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'admin', 'ResellerAdmin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:42:42.235066 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5eaf07c-89a4-48e4-a896-22018702f391 service nova] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:42:42.250242 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5eaf07c-89a4-48e4-a896-22018702f391 service nova] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:42:42.252339 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5eaf07c-89a4-48e4-a896-22018702f391 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:42:42.259701 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-d5eaf07c-89a4-48e4-a896-22018702f391 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:42:42.297718 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5eaf07c-89a4-48e4-a896-22018702f391 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:42:42.300778 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5eaf07c-89a4-48e4-a896-22018702f391 service nova] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:42:42.315528 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1377/2745] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 16:42:42 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 252 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 16:42:43.436216 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83c7e4d8-001a-4251-aea1-b30fb91bc035 None None] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:42:43.436698 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83c7e4d8-001a-4251-aea1-b30fb91bc035 None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:42:43.436788 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83c7e4d8-001a-4251-aea1-b30fb91bc035 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:42:43.457946 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.auth.core [None req-83c7e4d8-001a-4251-aea1-b30fb91bc035 None None] MFA Rules not processed for user `94f08e22c26f4c5d8a6789b1d7a0e1a7`. Rule list: `[]` (Enabled: `True`). {{(pid=81438) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:42:43.459194 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-83c7e4d8-001a-4251-aea1-b30fb91bc035 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:42:43.515840 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1369/2746] 10.222.0.50 () {58 vars in 958 bytes} [Tue Sep 13 16:42:43 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 81 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:42:43.762094 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5eaf07c-89a4-48e4-a896-22018702f391 service nova] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:42:43.772304 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5eaf07c-89a4-48e4-a896-22018702f391 service nova] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:42:43.779874 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bc71da2-f466-4b1b-b30f-05226b8cd6f7 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f436f07939d4855a21b195bc7566244', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:42:43.780499 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bc71da2-f466-4b1b-b30f-05226b8cd6f7 service neutron] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:42:43.780805 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bc71da2-f466-4b1b-b30f-05226b8cd6f7 service neutron] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:42:43.781075 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bc71da2-f466-4b1b-b30f-05226b8cd6f7 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:42:43.784090 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-1bc71da2-f466-4b1b-b30f-05226b8cd6f7 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:42:43.791929 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=100 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:42:43.794765 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1bc71da2-f466-4b1b-b30f-05226b8cd6f7 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:42:43.796604 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1bc71da2-f466-4b1b-b30f-05226b8cd6f7 service neutron] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:42:43.806504 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1378/2747] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 16:42:43 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 45 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 16:42:46.379235 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83c7e4d8-001a-4251-aea1-b30fb91bc035 None None] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:42:46.390069 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83c7e4d8-001a-4251-aea1-b30fb91bc035 None None] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:42:46.404757 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f603225-3fb2-451c-91a7-45878bfaa7e5 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f436f07939d4855a21b195bc7566244', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:42:46.405393 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f603225-3fb2-451c-91a7-45878bfaa7e5 service neutron] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:42:46.405812 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f603225-3fb2-451c-91a7-45878bfaa7e5 service neutron] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:42:46.406075 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f603225-3fb2-451c-91a7-45878bfaa7e5 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:42:46.410182 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-0f603225-3fb2-451c-91a7-45878bfaa7e5 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:42:46.426179 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f603225-3fb2-451c-91a7-45878bfaa7e5 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:42:46.427642 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f603225-3fb2-451c-91a7-45878bfaa7e5 service neutron] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:42:46.452866 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1370/2748] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 16:42:46 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 72 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 16:42:47.904910 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bc71da2-f466-4b1b-b30f-05226b8cd6f7 service neutron] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:42:47.913988 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bc71da2-f466-4b1b-b30f-05226b8cd6f7 service neutron] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:42:47.918168 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae04e2be-2699-43c1-8237-3c2b0f06b40d service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f436f07939d4855a21b195bc7566244', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:42:47.918763 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae04e2be-2699-43c1-8237-3c2b0f06b40d service neutron] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:42:47.919078 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae04e2be-2699-43c1-8237-3c2b0f06b40d service neutron] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:42:47.919353 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae04e2be-2699-43c1-8237-3c2b0f06b40d service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:42:47.921092 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-ae04e2be-2699-43c1-8237-3c2b0f06b40d service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:42:47.935106 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae04e2be-2699-43c1-8237-3c2b0f06b40d service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:42:47.936585 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae04e2be-2699-43c1-8237-3c2b0f06b40d service neutron] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:42:47.949875 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1379/2749] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 16:42:47 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 45 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 16:42:56.434989 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=86 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:42:57.943932 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=8 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:43:08.851549 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a91a531a-e24c-46a6-a5c1-faf534d1e3a9 None None] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:43:08.852055 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a91a531a-e24c-46a6-a5c1-faf534d1e3a9 None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:43:08.852164 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a91a531a-e24c-46a6-a5c1-faf534d1e3a9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:43:08.917713 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.auth.core [None req-a91a531a-e24c-46a6-a5c1-faf534d1e3a9 None None] MFA Rules not processed for user `f8364b1a190d4218bf353a89bdb81448`. Rule list: `[]` (Enabled: `True`). {{(pid=81438) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:43:08.918936 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-a91a531a-e24c-46a6-a5c1-faf534d1e3a9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:43:08.986509 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1371/2750] 10.222.0.50 () {58 vars in 1009 bytes} [Tue Sep 13 16:43:08 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 136 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:43:09.037625 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae04e2be-2699-43c1-8237-3c2b0f06b40d service neutron] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:43:09.039095 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-ae04e2be-2699-43c1-8237-3c2b0f06b40d service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:43:09.134115 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae04e2be-2699-43c1-8237-3c2b0f06b40d service neutron] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:43:09.139286 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c6d610b-1686-42e9-aca1-c94c84f78cb3 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b00ab0d6321d483188bc69deaddcdbbc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:43:09.139893 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c6d610b-1686-42e9-aca1-c94c84f78cb3 service swift] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:43:09.140175 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c6d610b-1686-42e9-aca1-c94c84f78cb3 service swift] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:43:09.140447 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c6d610b-1686-42e9-aca1-c94c84f78cb3 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:43:09.148670 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c6d610b-1686-42e9-aca1-c94c84f78cb3 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:43:09.150284 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c6d610b-1686-42e9-aca1-c94c84f78cb3 service swift] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:43:09.159545 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1380/2751] 10.222.0.50 () {60 vars in 1384 bytes} [Tue Sep 13 16:43:09 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 122 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Sep 13 16:43:09.955872 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a91a531a-e24c-46a6-a5c1-faf534d1e3a9 None None] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:43:10.031447 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a91a531a-e24c-46a6-a5c1-faf534d1e3a9 None None] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:43:10.036859 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af6972f8-59d2-4d15-92c1-4c43b2ae7782 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:43:10.037685 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af6972f8-59d2-4d15-92c1-4c43b2ae7782 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:43:10.038108 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af6972f8-59d2-4d15-92c1-4c43b2ae7782 admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:43:10.038543 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af6972f8-59d2-4d15-92c1-4c43b2ae7782 admin admin] PATH_INFO: `/v3/users/aad181425a954b31ab1ccf2f9657e2f2` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:43:10.039852 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af6972f8-59d2-4d15-92c1-4c43b2ae7782 admin admin] RBAC: Authorizing `identity:delete_user(user_id=aad181425a954b31ab1ccf2f9657e2f2)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:43:10.041824 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af6972f8-59d2-4d15-92c1-4c43b2ae7782 admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:43:10.098017 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-af6972f8-59d2-4d15-92c1-4c43b2ae7782 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'aad181425a954b31ab1ccf2f9657e2f2'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:43:10.106080 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-af6972f8-59d2-4d15-92c1-4c43b2ae7782 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'aad181425a954b31ab1ccf2f9657e2f2'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:43:10.117453 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=90,keystone:DELETE=9,keystone:INSERT=1 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:43:10.122761 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-af6972f8-59d2-4d15-92c1-4c43b2ae7782 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'aad181425a954b31ab1ccf2f9657e2f2'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:43:10.123688 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-af6972f8-59d2-4d15-92c1-4c43b2ae7782 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'aad181425a954b31ab1ccf2f9657e2f2'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:43:10.132694 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-af6972f8-59d2-4d15-92c1-4c43b2ae7782 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'aad181425a954b31ab1ccf2f9657e2f2'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:43:10.140125 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1372/2752] 10.222.0.50 () {58 vars in 1224 bytes} [Tue Sep 13 16:43:09 2022] DELETE /identity/v3/users/aad181425a954b31ab1ccf2f9657e2f2 => generated 0 bytes in 185 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 16:43:10.144599 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56906dd2-764e-4874-9b5a-a9b708178235 None None] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:43:10.145081 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56906dd2-764e-4874-9b5a-a9b708178235 None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:43:10.145294 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56906dd2-764e-4874-9b5a-a9b708178235 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:43:10.167902 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.auth.core [None req-56906dd2-764e-4874-9b5a-a9b708178235 None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81437) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:43:10.169280 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-56906dd2-764e-4874-9b5a-a9b708178235 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:43:10.241454 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1381/2753] 10.222.0.50 () {58 vars in 958 bytes} [Tue Sep 13 16:43:10 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 13 16:43:10.251881 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af6972f8-59d2-4d15-92c1-4c43b2ae7782 admin admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:43:10.253338 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-af6972f8-59d2-4d15-92c1-4c43b2ae7782 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:43:10.318851 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af6972f8-59d2-4d15-92c1-4c43b2ae7782 admin admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:43:10.324151 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-336f41fa-b455-488f-88b6-73abfdb9648d service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f436f07939d4855a21b195bc7566244', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:43:10.324854 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-336f41fa-b455-488f-88b6-73abfdb9648d service neutron] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:43:10.325202 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-336f41fa-b455-488f-88b6-73abfdb9648d service neutron] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:43:10.325562 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-336f41fa-b455-488f-88b6-73abfdb9648d service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:43:10.336931 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-336f41fa-b455-488f-88b6-73abfdb9648d service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:43:10.338496 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-336f41fa-b455-488f-88b6-73abfdb9648d service neutron] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:43:10.348580 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1373/2754] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 16:43:10 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 97 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 13 16:43:10.686046 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56906dd2-764e-4874-9b5a-a9b708178235 None None] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:43:10.687757 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-56906dd2-764e-4874-9b5a-a9b708178235 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:43:10.719179 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56906dd2-764e-4874-9b5a-a9b708178235 None None] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:43:10.726060 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-025d0945-6861-41f3-a29d-4350d6a83235 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:43:10.727054 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-025d0945-6861-41f3-a29d-4350d6a83235 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:43:10.727335 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-025d0945-6861-41f3-a29d-4350d6a83235 admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:43:10.727732 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-025d0945-6861-41f3-a29d-4350d6a83235 admin admin] PATH_INFO: `/v3/projects/8fafcd78e2fd4f5fb0eb12a8612ed1e0` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:43:10.729080 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-025d0945-6861-41f3-a29d-4350d6a83235 admin admin] RBAC: Authorizing `identity:delete_project(project_id=8fafcd78e2fd4f5fb0eb12a8612ed1e0)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:43:10.731173 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-025d0945-6861-41f3-a29d-4350d6a83235 admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:43:10.789590 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-025d0945-6861-41f3-a29d-4350d6a83235 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '8fafcd78e2fd4f5fb0eb12a8612ed1e0', 'request_id': 'req-025d0945-6861-41f3-a29d-4350d6a83235'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:43:10.800653 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-025d0945-6861-41f3-a29d-4350d6a83235 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '8fafcd78e2fd4f5fb0eb12a8612ed1e0', 'request_id': 'req-025d0945-6861-41f3-a29d-4350d6a83235'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:43:10.808602 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-025d0945-6861-41f3-a29d-4350d6a83235 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '8fafcd78e2fd4f5fb0eb12a8612ed1e0', 'request_id': 'req-025d0945-6861-41f3-a29d-4350d6a83235'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:43:10.814679 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=93,keystone:DELETE=6,keystone:INSERT=1 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:43:10.826068 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-025d0945-6861-41f3-a29d-4350d6a83235 admin admin] The token cache is being invalidate because project 8fafcd78e2fd4f5fb0eb12a8612ed1e0 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81437) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 13 16:43:10.826619 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-025d0945-6861-41f3-a29d-4350d6a83235 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:43:10.827653 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-025d0945-6861-41f3-a29d-4350d6a83235 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:43:10.830790 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1382/2755] 10.222.0.50 () {58 vars in 1233 bytes} [Tue Sep 13 16:43:10 2022] DELETE /identity/v3/projects/8fafcd78e2fd4f5fb0eb12a8612ed1e0 => generated 0 bytes in 145 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 16:43:10.835252 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3991d369-c392-4a4d-9041-1fb2c767c005 None None] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:43:10.835602 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3991d369-c392-4a4d-9041-1fb2c767c005 None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:43:10.835958 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3991d369-c392-4a4d-9041-1fb2c767c005 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:43:10.866285 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.auth.core [None req-3991d369-c392-4a4d-9041-1fb2c767c005 None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81438) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:43:10.867760 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-3991d369-c392-4a4d-9041-1fb2c767c005 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:43:10.932635 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1374/2756] 10.222.0.50 () {58 vars in 958 bytes} [Tue Sep 13 16:43:10 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:43:10.941742 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9b00097-2444-4282-8e49-749fbd6cd767 None None] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:43:10.942021 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9b00097-2444-4282-8e49-749fbd6cd767 None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:43:10.942343 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9b00097-2444-4282-8e49-749fbd6cd767 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:43:10.974520 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.auth.core [None req-a9b00097-2444-4282-8e49-749fbd6cd767 None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81437) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:43:10.975795 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-a9b00097-2444-4282-8e49-749fbd6cd767 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:43:10.989334 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1383/2757] 10.222.0.50 () {58 vars in 958 bytes} [Tue Sep 13 16:43:10 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 48 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:43:10.990495 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3991d369-c392-4a4d-9041-1fb2c767c005 None None] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:43:11.001619 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3991d369-c392-4a4d-9041-1fb2c767c005 None None] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:43:11.007218 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c7a0b87-814b-4c88-9ecb-e62fedf54f3c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:43:11.007844 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c7a0b87-814b-4c88-9ecb-e62fedf54f3c admin admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:43:11.008130 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c7a0b87-814b-4c88-9ecb-e62fedf54f3c admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:43:11.008401 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c7a0b87-814b-4c88-9ecb-e62fedf54f3c admin admin] PATH_INFO: `/v3/domains` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:43:11.009057 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c7a0b87-814b-4c88-9ecb-e62fedf54f3c admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81438) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 16:43:11.009362 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c7a0b87-814b-4c88-9ecb-e62fedf54f3c admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:43:11.011190 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c7a0b87-814b-4c88-9ecb-e62fedf54f3c admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:43:11.027048 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1375/2758] 10.222.0.50 () {58 vars in 1153 bytes} [Tue Sep 13 16:43:10 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 16:43:11.033801 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9b00097-2444-4282-8e49-749fbd6cd767 None None] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:43:11.047880 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9b00097-2444-4282-8e49-749fbd6cd767 None None] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:43:11.055396 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21d1aa96-7018-421f-978f-4b7909330b48 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:43:11.056017 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21d1aa96-7018-421f-978f-4b7909330b48 admin admin] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:43:11.056292 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21d1aa96-7018-421f-978f-4b7909330b48 admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:43:11.056560 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21d1aa96-7018-421f-978f-4b7909330b48 admin admin] PATH_INFO: `/v3/projects` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:43:11.057649 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21d1aa96-7018-421f-978f-4b7909330b48 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:43:11.060017 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21d1aa96-7018-421f-978f-4b7909330b48 admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:43:11.105540 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1384/2759] 10.222.0.50 () {60 vars in 1153 bytes} [Tue Sep 13 16:43:11 2022] POST /identity/v3/projects => generated 375 bytes in 73 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 16:43:11.106078 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c7a0b87-814b-4c88-9ecb-e62fedf54f3c admin admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:43:11.123482 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c7a0b87-814b-4c88-9ecb-e62fedf54f3c admin admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:43:11.202764 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b8e86f0-27ff-4f27-8605-ed9824118b7c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:43:11.203457 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b8e86f0-27ff-4f27-8605-ed9824118b7c admin admin] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:43:11.203740 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b8e86f0-27ff-4f27-8605-ed9824118b7c admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:43:11.204018 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b8e86f0-27ff-4f27-8605-ed9824118b7c admin admin] PATH_INFO: `/v3/users` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:43:11.205598 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b8e86f0-27ff-4f27-8605-ed9824118b7c admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:43:11.207951 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b8e86f0-27ff-4f27-8605-ed9824118b7c admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:43:11.233604 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1376/2760] 10.222.0.50 () {60 vars in 1144 bytes} [Tue Sep 13 16:43:11 2022] POST /identity/v3/users => generated 288 bytes in 128 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 13 16:43:11.239142 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21d1aa96-7018-421f-978f-4b7909330b48 admin admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:43:11.253311 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21d1aa96-7018-421f-978f-4b7909330b48 admin admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:43:11.262821 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e35e4ba-2649-4698-87b2-b7b67fe1ab37 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:43:11.264316 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e35e4ba-2649-4698-87b2-b7b67fe1ab37 admin admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:43:11.264857 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e35e4ba-2649-4698-87b2-b7b67fe1ab37 admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:43:11.265411 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e35e4ba-2649-4698-87b2-b7b67fe1ab37 admin admin] PATH_INFO: `/v3/roles` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:43:11.266384 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e35e4ba-2649-4698-87b2-b7b67fe1ab37 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:43:11.270438 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e35e4ba-2649-4698-87b2-b7b67fe1ab37 admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:43:11.285255 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1385/2761] 10.222.0.50 () {58 vars in 1122 bytes} [Tue Sep 13 16:43:11 2022] GET /identity/v3/roles => generated 1420 bytes in 47 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 16:43:11.290724 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b8e86f0-27ff-4f27-8605-ed9824118b7c admin admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:43:11.296286 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=96,keystone:DELETE=1,keystone:INSERT=3 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:43:11.304330 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b8e86f0-27ff-4f27-8605-ed9824118b7c admin admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:43:11.315097 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40b676dd-28c4-4d13-8000-9677509866e0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:43:11.317480 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40b676dd-28c4-4d13-8000-9677509866e0 admin admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:43:11.317480 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40b676dd-28c4-4d13-8000-9677509866e0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:43:11.317480 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40b676dd-28c4-4d13-8000-9677509866e0 admin admin] PATH_INFO: `/v3/roles` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:43:11.317480 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40b676dd-28c4-4d13-8000-9677509866e0 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:43:11.318982 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40b676dd-28c4-4d13-8000-9677509866e0 admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:43:11.333566 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1377/2762] 10.222.0.50 () {58 vars in 1122 bytes} [Tue Sep 13 16:43:11 2022] GET /identity/v3/roles => generated 1420 bytes in 43 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 16:43:11.338986 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e35e4ba-2649-4698-87b2-b7b67fe1ab37 admin admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:43:11.350073 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e35e4ba-2649-4698-87b2-b7b67fe1ab37 admin admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:43:11.356457 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb88a65d-d218-4e9a-b148-896059625554 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:43:11.356457 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb88a65d-d218-4e9a-b148-896059625554 admin admin] REQUEST_METHOD: `PUT` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:43:11.356457 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb88a65d-d218-4e9a-b148-896059625554 admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:43:11.356951 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb88a65d-d218-4e9a-b148-896059625554 admin admin] PATH_INFO: `/v3/projects/d5fbc65c758d45fb8a0f52dc42e6e1dd/users/23c7b91d5e8f44e7b710ac9374559d59/roles/15db4aaca3264ae29bde38903ba43471` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:43:11.373203 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb88a65d-d218-4e9a-b148-896059625554 admin admin] RBAC: Authorizing `identity:create_grant(project_id=d5fbc65c758d45fb8a0f52dc42e6e1dd, user_id=23c7b91d5e8f44e7b710ac9374559d59, role_id=15db4aaca3264ae29bde38903ba43471)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:43:11.375968 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb88a65d-d218-4e9a-b148-896059625554 admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:43:11.395162 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1386/2763] 10.222.0.50 () {60 vars in 1483 bytes} [Tue Sep 13 16:43:11 2022] PUT /identity/v3/projects/d5fbc65c758d45fb8a0f52dc42e6e1dd/users/23c7b91d5e8f44e7b710ac9374559d59/roles/15db4aaca3264ae29bde38903ba43471 => generated 0 bytes in 57 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 16:43:11.406507 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdc87df6-748b-4489-a085-cf0372952374 None None] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:43:11.407490 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdc87df6-748b-4489-a085-cf0372952374 None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:43:11.407999 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdc87df6-748b-4489-a085-cf0372952374 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:43:11.445842 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.auth.core [None req-fdc87df6-748b-4489-a085-cf0372952374 None None] MFA Rules not processed for user `23c7b91d5e8f44e7b710ac9374559d59`. Rule list: `[]` (Enabled: `True`). {{(pid=81438) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:43:11.447118 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-fdc87df6-748b-4489-a085-cf0372952374 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:43:11.531155 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1378/2764] 10.222.0.50 () {58 vars in 958 bytes} [Tue Sep 13 16:43:11 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 127 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:43:11.543367 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb88a65d-d218-4e9a-b148-896059625554 admin admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:43:11.545172 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-fb88a65d-d218-4e9a-b148-896059625554 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:43:11.651842 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb88a65d-d218-4e9a-b148-896059625554 admin admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:43:11.660140 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf988c92-9eff-4032-aab0-19708070507e service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7fd6b5afd67b48f9b0f0597ccd4cf4df', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:43:11.660758 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf988c92-9eff-4032-aab0-19708070507e service cinder] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:43:11.661038 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf988c92-9eff-4032-aab0-19708070507e service cinder] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:43:11.661310 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf988c92-9eff-4032-aab0-19708070507e service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:43:11.672160 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf988c92-9eff-4032-aab0-19708070507e service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:43:11.675002 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf988c92-9eff-4032-aab0-19708070507e service cinder] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:43:11.687131 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1387/2765] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 16:43:11 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 144 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 16:43:21.525749 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=36 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:43:21.681444 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:DELETE=1,keystone:SELECT=55,keystone:INSERT=2 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:43:22.432010 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdc87df6-748b-4489-a085-cf0372952374 None None] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:43:22.502481 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdc87df6-748b-4489-a085-cf0372952374 None None] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:43:22.507470 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c0fd032-a445-4f78-a36a-b4d7a8f86f6e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:43:22.508545 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c0fd032-a445-4f78-a36a-b4d7a8f86f6e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:43:22.509004 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c0fd032-a445-4f78-a36a-b4d7a8f86f6e admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:43:22.509640 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c0fd032-a445-4f78-a36a-b4d7a8f86f6e admin admin] PATH_INFO: `/v3/users/23c7b91d5e8f44e7b710ac9374559d59` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:43:22.511431 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c0fd032-a445-4f78-a36a-b4d7a8f86f6e admin admin] RBAC: Authorizing `identity:delete_user(user_id=23c7b91d5e8f44e7b710ac9374559d59)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:43:22.513880 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c0fd032-a445-4f78-a36a-b4d7a8f86f6e admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:43:22.574122 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-5c0fd032-a445-4f78-a36a-b4d7a8f86f6e admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '23c7b91d5e8f44e7b710ac9374559d59'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:43:22.585696 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-5c0fd032-a445-4f78-a36a-b4d7a8f86f6e admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '23c7b91d5e8f44e7b710ac9374559d59'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:43:22.599149 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-5c0fd032-a445-4f78-a36a-b4d7a8f86f6e admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '23c7b91d5e8f44e7b710ac9374559d59'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:43:22.600249 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-5c0fd032-a445-4f78-a36a-b4d7a8f86f6e admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '23c7b91d5e8f44e7b710ac9374559d59'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:43:22.612189 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-5c0fd032-a445-4f78-a36a-b4d7a8f86f6e admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '23c7b91d5e8f44e7b710ac9374559d59'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:43:22.618938 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1379/2766] 10.222.0.50 () {58 vars in 1224 bytes} [Tue Sep 13 16:43:22 2022] DELETE /identity/v3/users/23c7b91d5e8f44e7b710ac9374559d59 => generated 0 bytes in 187 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 16:43:22.623459 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8402d979-0e2c-4560-b8c9-13afa42f5b1f None None] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:43:22.623979 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8402d979-0e2c-4560-b8c9-13afa42f5b1f None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:43:22.624295 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8402d979-0e2c-4560-b8c9-13afa42f5b1f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:43:22.645692 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.auth.core [None req-8402d979-0e2c-4560-b8c9-13afa42f5b1f None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81437) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:43:22.647252 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-8402d979-0e2c-4560-b8c9-13afa42f5b1f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:43:22.730538 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1388/2767] 10.222.0.50 () {58 vars in 958 bytes} [Tue Sep 13 16:43:22 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:43:22.737373 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c0fd032-a445-4f78-a36a-b4d7a8f86f6e admin admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:43:22.738845 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-5c0fd032-a445-4f78-a36a-b4d7a8f86f6e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:43:22.824526 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c0fd032-a445-4f78-a36a-b4d7a8f86f6e admin admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:43:22.831673 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6239305a-cb76-4253-a080-363610918e05 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f436f07939d4855a21b195bc7566244', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:43:22.832367 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6239305a-cb76-4253-a080-363610918e05 service neutron] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:43:22.832665 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6239305a-cb76-4253-a080-363610918e05 service neutron] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:43:22.833085 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6239305a-cb76-4253-a080-363610918e05 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:43:22.846028 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6239305a-cb76-4253-a080-363610918e05 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:43:22.847696 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6239305a-cb76-4253-a080-363610918e05 service neutron] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:43:22.858705 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1380/2768] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 16:43:22 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 122 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 16:43:23.230199 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8402d979-0e2c-4560-b8c9-13afa42f5b1f None None] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:43:23.232326 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-8402d979-0e2c-4560-b8c9-13afa42f5b1f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:43:23.259514 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8402d979-0e2c-4560-b8c9-13afa42f5b1f None None] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:43:23.270587 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea62c8a5-3eef-493f-9645-86a712421348 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:43:23.271691 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea62c8a5-3eef-493f-9645-86a712421348 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:43:23.272668 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea62c8a5-3eef-493f-9645-86a712421348 admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:43:23.273019 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea62c8a5-3eef-493f-9645-86a712421348 admin admin] PATH_INFO: `/v3/projects/d5fbc65c758d45fb8a0f52dc42e6e1dd` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:43:23.274579 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea62c8a5-3eef-493f-9645-86a712421348 admin admin] RBAC: Authorizing `identity:delete_project(project_id=d5fbc65c758d45fb8a0f52dc42e6e1dd)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:43:23.277344 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea62c8a5-3eef-493f-9645-86a712421348 admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:43:23.342335 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-ea62c8a5-3eef-493f-9645-86a712421348 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'd5fbc65c758d45fb8a0f52dc42e6e1dd', 'request_id': 'req-ea62c8a5-3eef-493f-9645-86a712421348'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:43:23.353008 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-ea62c8a5-3eef-493f-9645-86a712421348 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'd5fbc65c758d45fb8a0f52dc42e6e1dd', 'request_id': 'req-ea62c8a5-3eef-493f-9645-86a712421348'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:43:23.362192 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-ea62c8a5-3eef-493f-9645-86a712421348 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'd5fbc65c758d45fb8a0f52dc42e6e1dd', 'request_id': 'req-ea62c8a5-3eef-493f-9645-86a712421348'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:43:23.379093 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-ea62c8a5-3eef-493f-9645-86a712421348 admin admin] The token cache is being invalidate because project d5fbc65c758d45fb8a0f52dc42e6e1dd was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81437) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 13 16:43:23.379711 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-ea62c8a5-3eef-493f-9645-86a712421348 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:43:23.380730 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-ea62c8a5-3eef-493f-9645-86a712421348 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:43:23.382371 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1389/2769] 10.222.0.50 () {58 vars in 1233 bytes} [Tue Sep 13 16:43:23 2022] DELETE /identity/v3/projects/d5fbc65c758d45fb8a0f52dc42e6e1dd => generated 0 bytes in 153 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 16:43:25.346817 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a709e04-6733-4307-b05c-2c55a6ecf958 None None] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:43:25.347339 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a709e04-6733-4307-b05c-2c55a6ecf958 None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:43:25.347446 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a709e04-6733-4307-b05c-2c55a6ecf958 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:43:25.366231 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.auth.core [None req-8a709e04-6733-4307-b05c-2c55a6ecf958 None None] MFA Rules not processed for user `f8364b1a190d4218bf353a89bdb81448`. Rule list: `[]` (Enabled: `True`). {{(pid=81438) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:43:25.367484 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-8a709e04-6733-4307-b05c-2c55a6ecf958 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:43:25.383851 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:43:25.426259 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1381/2770] 10.222.0.50 () {58 vars in 967 bytes} [Tue Sep 13 16:43:25 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 80 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:43:25.438798 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea62c8a5-3eef-493f-9645-86a712421348 admin admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:43:25.440244 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-ea62c8a5-3eef-493f-9645-86a712421348 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:43:25.508340 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea62c8a5-3eef-493f-9645-86a712421348 admin admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:43:25.513468 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3b30023-4674-4260-b413-41a25b8e6e1c service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b00ab0d6321d483188bc69deaddcdbbc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:43:25.514082 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3b30023-4674-4260-b413-41a25b8e6e1c service swift] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:43:25.514365 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3b30023-4674-4260-b413-41a25b8e6e1c service swift] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:43:25.514640 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3b30023-4674-4260-b413-41a25b8e6e1c service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:43:25.523583 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3b30023-4674-4260-b413-41a25b8e6e1c service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:43:25.525151 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3b30023-4674-4260-b413-41a25b8e6e1c service swift] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:43:25.533861 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1390/2771] 10.222.0.50 () {60 vars in 1384 bytes} [Tue Sep 13 16:43:25 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 96 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Sep 13 16:43:27.452221 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a709e04-6733-4307-b05c-2c55a6ecf958 None None] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:43:27.453717 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-8a709e04-6733-4307-b05c-2c55a6ecf958 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:43:27.522103 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a709e04-6733-4307-b05c-2c55a6ecf958 None None] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:43:27.526699 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22278d1d-6879-4f7d-96f4-78b59b732b56 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:43:27.527361 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22278d1d-6879-4f7d-96f4-78b59b732b56 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:43:27.527643 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22278d1d-6879-4f7d-96f4-78b59b732b56 admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:43:27.527918 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22278d1d-6879-4f7d-96f4-78b59b732b56 admin admin] PATH_INFO: `/v3/users/94f08e22c26f4c5d8a6789b1d7a0e1a7` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:43:27.529047 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22278d1d-6879-4f7d-96f4-78b59b732b56 admin admin] RBAC: Authorizing `identity:delete_user(user_id=94f08e22c26f4c5d8a6789b1d7a0e1a7)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:43:27.530975 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22278d1d-6879-4f7d-96f4-78b59b732b56 admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:43:27.579027 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-22278d1d-6879-4f7d-96f4-78b59b732b56 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '94f08e22c26f4c5d8a6789b1d7a0e1a7'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:43:27.587455 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-22278d1d-6879-4f7d-96f4-78b59b732b56 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '94f08e22c26f4c5d8a6789b1d7a0e1a7'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:43:27.600085 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-22278d1d-6879-4f7d-96f4-78b59b732b56 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '94f08e22c26f4c5d8a6789b1d7a0e1a7'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:43:27.600723 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-22278d1d-6879-4f7d-96f4-78b59b732b56 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '94f08e22c26f4c5d8a6789b1d7a0e1a7'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:43:27.607624 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-22278d1d-6879-4f7d-96f4-78b59b732b56 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '94f08e22c26f4c5d8a6789b1d7a0e1a7'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:43:27.613474 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1382/2772] 10.222.0.50 () {58 vars in 1224 bytes} [Tue Sep 13 16:43:27 2022] DELETE /identity/v3/users/94f08e22c26f4c5d8a6789b1d7a0e1a7 => generated 0 bytes in 162 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 16:43:27.785973 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3b30023-4674-4260-b413-41a25b8e6e1c service swift] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:43:27.796467 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3b30023-4674-4260-b413-41a25b8e6e1c service swift] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:43:27.808659 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:43:27.860459 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b84a4840-19d4-49fe-bf02-cacbcf4fd297 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:43:27.861083 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b84a4840-19d4-49fe-bf02-cacbcf4fd297 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:43:27.861361 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b84a4840-19d4-49fe-bf02-cacbcf4fd297 admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:43:27.861717 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b84a4840-19d4-49fe-bf02-cacbcf4fd297 admin admin] PATH_INFO: `/v3/projects/b70b7595c0e44310ba172e35bc6e8fe3` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:43:27.862827 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b84a4840-19d4-49fe-bf02-cacbcf4fd297 admin admin] RBAC: Authorizing `identity:delete_project(project_id=b70b7595c0e44310ba172e35bc6e8fe3)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:43:27.865079 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b84a4840-19d4-49fe-bf02-cacbcf4fd297 admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:43:27.915973 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-b84a4840-19d4-49fe-bf02-cacbcf4fd297 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'b70b7595c0e44310ba172e35bc6e8fe3', 'request_id': 'req-b84a4840-19d4-49fe-bf02-cacbcf4fd297'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:43:27.923331 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-b84a4840-19d4-49fe-bf02-cacbcf4fd297 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'b70b7595c0e44310ba172e35bc6e8fe3', 'request_id': 'req-b84a4840-19d4-49fe-bf02-cacbcf4fd297'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:43:27.929684 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-b84a4840-19d4-49fe-bf02-cacbcf4fd297 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'b70b7595c0e44310ba172e35bc6e8fe3', 'request_id': 'req-b84a4840-19d4-49fe-bf02-cacbcf4fd297'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:43:27.940837 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-b84a4840-19d4-49fe-bf02-cacbcf4fd297 admin admin] The token cache is being invalidate because project b70b7595c0e44310ba172e35bc6e8fe3 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81437) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 13 16:43:27.941121 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-b84a4840-19d4-49fe-bf02-cacbcf4fd297 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:43:27.941760 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-b84a4840-19d4-49fe-bf02-cacbcf4fd297 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:43:27.943273 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1391/2773] 10.222.0.50 () {58 vars in 1233 bytes} [Tue Sep 13 16:43:27 2022] DELETE /identity/v3/projects/b70b7595c0e44310ba172e35bc6e8fe3 => generated 0 bytes in 158 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 13 16:43:27.948322 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74f5c01b-f8ed-4d20-88ed-09652d2ed011 None None] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:43:27.948659 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74f5c01b-f8ed-4d20-88ed-09652d2ed011 None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:43:27.948976 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74f5c01b-f8ed-4d20-88ed-09652d2ed011 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:43:27.969218 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.auth.core [None req-74f5c01b-f8ed-4d20-88ed-09652d2ed011 None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81438) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:43:27.970357 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-74f5c01b-f8ed-4d20-88ed-09652d2ed011 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:43:28.012191 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:43:28.032446 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1383/2774] 10.222.0.50 () {58 vars in 958 bytes} [Tue Sep 13 16:43:27 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:43:28.040623 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-566579d3-588d-4f42-bd47-85fb71a89ebb None None] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:43:28.040906 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-566579d3-588d-4f42-bd47-85fb71a89ebb None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:43:28.041184 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-566579d3-588d-4f42-bd47-85fb71a89ebb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:43:28.062370 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.auth.core [None req-566579d3-588d-4f42-bd47-85fb71a89ebb None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81437) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:43:28.063618 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-566579d3-588d-4f42-bd47-85fb71a89ebb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:43:28.072849 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1392/2775] 10.222.0.50 () {58 vars in 958 bytes} [Tue Sep 13 16:43:28 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:43:28.077195 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74f5c01b-f8ed-4d20-88ed-09652d2ed011 None None] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:43:28.087750 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74f5c01b-f8ed-4d20-88ed-09652d2ed011 None None] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:43:28.092242 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-448a2c34-4968-48b3-95c2-40e067929cf0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:43:28.092838 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-448a2c34-4968-48b3-95c2-40e067929cf0 admin admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:43:28.093121 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-448a2c34-4968-48b3-95c2-40e067929cf0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:43:28.093391 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-448a2c34-4968-48b3-95c2-40e067929cf0 admin admin] PATH_INFO: `/v3/domains` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:43:28.094025 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-448a2c34-4968-48b3-95c2-40e067929cf0 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81438) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 16:43:28.094325 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-448a2c34-4968-48b3-95c2-40e067929cf0 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:43:28.095949 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-448a2c34-4968-48b3-95c2-40e067929cf0 admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:43:28.107881 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1384/2776] 10.222.0.50 () {58 vars in 1153 bytes} [Tue Sep 13 16:43:28 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 16:43:28.112652 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-566579d3-588d-4f42-bd47-85fb71a89ebb None None] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:43:28.121769 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-566579d3-588d-4f42-bd47-85fb71a89ebb None None] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:43:28.126432 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8daed7af-b76f-4759-8693-c6ec15eeedff admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:43:28.127062 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8daed7af-b76f-4759-8693-c6ec15eeedff admin admin] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:43:28.127880 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8daed7af-b76f-4759-8693-c6ec15eeedff admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:43:28.128153 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8daed7af-b76f-4759-8693-c6ec15eeedff admin admin] PATH_INFO: `/v3/projects` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:43:28.128777 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8daed7af-b76f-4759-8693-c6ec15eeedff admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:43:28.130677 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8daed7af-b76f-4759-8693-c6ec15eeedff admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:43:28.157798 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1393/2777] 10.222.0.50 () {60 vars in 1153 bytes} [Tue Sep 13 16:43:28 2022] POST /identity/v3/projects => generated 373 bytes in 46 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 16:43:28.162123 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-448a2c34-4968-48b3-95c2-40e067929cf0 admin admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:43:28.171888 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-448a2c34-4968-48b3-95c2-40e067929cf0 admin admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:43:28.226855 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9685ac5f-eb16-43cb-a569-5f06a23e0099 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:43:28.227407 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9685ac5f-eb16-43cb-a569-5f06a23e0099 admin admin] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:43:28.227684 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9685ac5f-eb16-43cb-a569-5f06a23e0099 admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:43:28.227954 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9685ac5f-eb16-43cb-a569-5f06a23e0099 admin admin] PATH_INFO: `/v3/users` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:43:28.228569 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9685ac5f-eb16-43cb-a569-5f06a23e0099 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:43:28.230481 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9685ac5f-eb16-43cb-a569-5f06a23e0099 admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:43:28.251406 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1385/2778] 10.222.0.50 () {60 vars in 1144 bytes} [Tue Sep 13 16:43:28 2022] POST /identity/v3/users => generated 287 bytes in 90 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 13 16:43:28.256182 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8daed7af-b76f-4759-8693-c6ec15eeedff admin admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:43:28.266729 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8daed7af-b76f-4759-8693-c6ec15eeedff admin admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:43:28.271455 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7abacb8b-e955-4d78-9242-8d792c6f3d68 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:43:28.272045 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7abacb8b-e955-4d78-9242-8d792c6f3d68 admin admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:43:28.272322 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7abacb8b-e955-4d78-9242-8d792c6f3d68 admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:43:28.272592 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7abacb8b-e955-4d78-9242-8d792c6f3d68 admin admin] PATH_INFO: `/v3/roles` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:43:28.273223 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7abacb8b-e955-4d78-9242-8d792c6f3d68 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:43:28.274817 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7abacb8b-e955-4d78-9242-8d792c6f3d68 admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:43:28.285293 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1394/2779] 10.222.0.50 () {58 vars in 1122 bytes} [Tue Sep 13 16:43:28 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 16:43:28.289192 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9685ac5f-eb16-43cb-a569-5f06a23e0099 admin admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:43:28.299719 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9685ac5f-eb16-43cb-a569-5f06a23e0099 admin admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:43:28.304818 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fe002fd-e752-4482-b580-4c2c27cedaf4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:43:28.305411 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fe002fd-e752-4482-b580-4c2c27cedaf4 admin admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:43:28.305694 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fe002fd-e752-4482-b580-4c2c27cedaf4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:43:28.305970 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fe002fd-e752-4482-b580-4c2c27cedaf4 admin admin] PATH_INFO: `/v3/roles` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:43:28.306607 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fe002fd-e752-4482-b580-4c2c27cedaf4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:43:28.308234 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fe002fd-e752-4482-b580-4c2c27cedaf4 admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:43:28.319380 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1386/2780] 10.222.0.50 () {58 vars in 1122 bytes} [Tue Sep 13 16:43:28 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 16:43:28.323380 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7abacb8b-e955-4d78-9242-8d792c6f3d68 admin admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:43:28.334105 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7abacb8b-e955-4d78-9242-8d792c6f3d68 admin admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:43:28.338861 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5932c51e-f2c6-4d28-b9bb-e03f8462c4e3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:43:28.339531 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5932c51e-f2c6-4d28-b9bb-e03f8462c4e3 admin admin] REQUEST_METHOD: `PUT` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:43:28.339825 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5932c51e-f2c6-4d28-b9bb-e03f8462c4e3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:43:28.340090 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5932c51e-f2c6-4d28-b9bb-e03f8462c4e3 admin admin] PATH_INFO: `/v3/projects/6e09d443c9144e0f8be91b145dd11b9c/users/ad35243edcf44f828ebe89af8921cea2/roles/15db4aaca3264ae29bde38903ba43471` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:43:28.355344 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5932c51e-f2c6-4d28-b9bb-e03f8462c4e3 admin admin] RBAC: Authorizing `identity:create_grant(project_id=6e09d443c9144e0f8be91b145dd11b9c, user_id=ad35243edcf44f828ebe89af8921cea2, role_id=15db4aaca3264ae29bde38903ba43471)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:43:28.357994 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5932c51e-f2c6-4d28-b9bb-e03f8462c4e3 admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:43:28.371096 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1395/2781] 10.222.0.50 () {60 vars in 1483 bytes} [Tue Sep 13 16:43:28 2022] PUT /identity/v3/projects/6e09d443c9144e0f8be91b145dd11b9c/users/ad35243edcf44f828ebe89af8921cea2/roles/15db4aaca3264ae29bde38903ba43471 => generated 0 bytes in 48 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 16:43:28.380920 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dcf27d2-0dab-4417-81d4-41e42ebe135c None None] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:43:28.381211 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dcf27d2-0dab-4417-81d4-41e42ebe135c None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:43:28.381487 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dcf27d2-0dab-4417-81d4-41e42ebe135c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:43:28.409720 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.auth.core [None req-6dcf27d2-0dab-4417-81d4-41e42ebe135c None None] MFA Rules not processed for user `ad35243edcf44f828ebe89af8921cea2`. Rule list: `[]` (Enabled: `True`). {{(pid=81438) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:43:28.410938 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-6dcf27d2-0dab-4417-81d4-41e42ebe135c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:43:28.474691 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1387/2782] 10.222.0.50 () {58 vars in 958 bytes} [Tue Sep 13 16:43:28 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:43:28.485927 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5932c51e-f2c6-4d28-b9bb-e03f8462c4e3 admin admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:43:28.488082 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-5932c51e-f2c6-4d28-b9bb-e03f8462c4e3 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:43:28.531816 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:43:28.547350 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5932c51e-f2c6-4d28-b9bb-e03f8462c4e3 admin admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:43:28.551190 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6967137-6e3a-4695-8499-0a44d3aa0d17 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7fd6b5afd67b48f9b0f0597ccd4cf4df', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:43:28.551793 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6967137-6e3a-4695-8499-0a44d3aa0d17 service cinder] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:43:28.552408 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6967137-6e3a-4695-8499-0a44d3aa0d17 service cinder] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:43:28.552408 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6967137-6e3a-4695-8499-0a44d3aa0d17 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:43:28.560843 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6967137-6e3a-4695-8499-0a44d3aa0d17 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:43:28.562390 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6967137-6e3a-4695-8499-0a44d3aa0d17 service cinder] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:43:28.572234 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1396/2783] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 16:43:28 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 87 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 16:43:28.599529 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6dcf27d2-0dab-4417-81d4-41e42ebe135c None None] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:43:28.642891 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:43:28.662059 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6dcf27d2-0dab-4417-81d4-41e42ebe135c None None] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:43:28.666724 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cb454d3-07aa-4e9d-84d7-b32db5185218 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:43:28.667389 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cb454d3-07aa-4e9d-84d7-b32db5185218 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:43:28.667670 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cb454d3-07aa-4e9d-84d7-b32db5185218 admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:43:28.667944 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cb454d3-07aa-4e9d-84d7-b32db5185218 admin admin] PATH_INFO: `/v3/users/ad35243edcf44f828ebe89af8921cea2` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:43:28.669167 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9cb454d3-07aa-4e9d-84d7-b32db5185218 admin admin] RBAC: Authorizing `identity:delete_user(user_id=ad35243edcf44f828ebe89af8921cea2)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:43:28.671106 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9cb454d3-07aa-4e9d-84d7-b32db5185218 admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:43:28.717780 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-9cb454d3-07aa-4e9d-84d7-b32db5185218 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'ad35243edcf44f828ebe89af8921cea2'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:43:28.725677 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-9cb454d3-07aa-4e9d-84d7-b32db5185218 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'ad35243edcf44f828ebe89af8921cea2'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:43:28.737914 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-9cb454d3-07aa-4e9d-84d7-b32db5185218 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'ad35243edcf44f828ebe89af8921cea2'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:43:28.738543 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-9cb454d3-07aa-4e9d-84d7-b32db5185218 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'ad35243edcf44f828ebe89af8921cea2'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:43:28.745339 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-9cb454d3-07aa-4e9d-84d7-b32db5185218 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'ad35243edcf44f828ebe89af8921cea2'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:43:28.751147 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1388/2784] 10.222.0.50 () {58 vars in 1224 bytes} [Tue Sep 13 16:43:28 2022] DELETE /identity/v3/users/ad35243edcf44f828ebe89af8921cea2 => generated 0 bytes in 152 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 16:43:28.756290 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b35952c-2fa7-404c-bd62-f3863c56b69c None None] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:43:28.756574 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b35952c-2fa7-404c-bd62-f3863c56b69c None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:43:28.756848 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b35952c-2fa7-404c-bd62-f3863c56b69c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:43:28.776412 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.auth.core [None req-2b35952c-2fa7-404c-bd62-f3863c56b69c None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81437) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:43:28.777594 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-2b35952c-2fa7-404c-bd62-f3863c56b69c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:43:28.835573 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1397/2785] 10.222.0.50 () {58 vars in 958 bytes} [Tue Sep 13 16:43:28 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 80 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:43:28.844983 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cb454d3-07aa-4e9d-84d7-b32db5185218 admin admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:43:28.846527 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-9cb454d3-07aa-4e9d-84d7-b32db5185218 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:43:28.914450 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cb454d3-07aa-4e9d-84d7-b32db5185218 admin admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:43:28.918112 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-902032a5-277f-4b3d-9b0f-2c0b9c18c2ff service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f436f07939d4855a21b195bc7566244', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:43:28.918716 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-902032a5-277f-4b3d-9b0f-2c0b9c18c2ff service neutron] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:43:28.919024 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-902032a5-277f-4b3d-9b0f-2c0b9c18c2ff service neutron] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:43:28.919299 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-902032a5-277f-4b3d-9b0f-2c0b9c18c2ff service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:43:28.929323 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-902032a5-277f-4b3d-9b0f-2c0b9c18c2ff service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:43:28.930769 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-902032a5-277f-4b3d-9b0f-2c0b9c18c2ff service neutron] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:43:28.940949 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1389/2786] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 16:43:28 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 96 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 16:43:29.211361 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b35952c-2fa7-404c-bd62-f3863c56b69c None None] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:43:29.212842 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-2b35952c-2fa7-404c-bd62-f3863c56b69c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:43:29.225212 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b35952c-2fa7-404c-bd62-f3863c56b69c None None] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:43:29.230047 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ff9688e-e3a3-4c6d-b20f-493bd2e5bf8b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:43:29.230684 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ff9688e-e3a3-4c6d-b20f-493bd2e5bf8b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:43:29.230994 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ff9688e-e3a3-4c6d-b20f-493bd2e5bf8b admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:43:29.231283 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ff9688e-e3a3-4c6d-b20f-493bd2e5bf8b admin admin] PATH_INFO: `/v3/projects/6e09d443c9144e0f8be91b145dd11b9c` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:43:29.232403 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ff9688e-e3a3-4c6d-b20f-493bd2e5bf8b admin admin] RBAC: Authorizing `identity:delete_project(project_id=6e09d443c9144e0f8be91b145dd11b9c)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:43:29.234333 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ff9688e-e3a3-4c6d-b20f-493bd2e5bf8b admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:43:29.281887 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-1ff9688e-e3a3-4c6d-b20f-493bd2e5bf8b admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '6e09d443c9144e0f8be91b145dd11b9c', 'request_id': 'req-1ff9688e-e3a3-4c6d-b20f-493bd2e5bf8b'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:43:29.289112 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-1ff9688e-e3a3-4c6d-b20f-493bd2e5bf8b admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '6e09d443c9144e0f8be91b145dd11b9c', 'request_id': 'req-1ff9688e-e3a3-4c6d-b20f-493bd2e5bf8b'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:43:29.295350 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-1ff9688e-e3a3-4c6d-b20f-493bd2e5bf8b admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '6e09d443c9144e0f8be91b145dd11b9c', 'request_id': 'req-1ff9688e-e3a3-4c6d-b20f-493bd2e5bf8b'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:43:29.306301 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-1ff9688e-e3a3-4c6d-b20f-493bd2e5bf8b admin admin] The token cache is being invalidate because project 6e09d443c9144e0f8be91b145dd11b9c was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81437) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 13 16:43:29.306591 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-1ff9688e-e3a3-4c6d-b20f-493bd2e5bf8b admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:43:29.307239 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-1ff9688e-e3a3-4c6d-b20f-493bd2e5bf8b admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:43:29.308712 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1398/2787] 10.222.0.50 () {58 vars in 1233 bytes} [Tue Sep 13 16:43:29 2022] DELETE /identity/v3/projects/6e09d443c9144e0f8be91b145dd11b9c => generated 0 bytes in 98 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 16:43:29.313637 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2e0bd85-5c23-4f82-8af0-8fc9618ca287 None None] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:43:29.313924 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2e0bd85-5c23-4f82-8af0-8fc9618ca287 None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:43:29.314194 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2e0bd85-5c23-4f82-8af0-8fc9618ca287 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:43:29.335759 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.auth.core [None req-c2e0bd85-5c23-4f82-8af0-8fc9618ca287 None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81438) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:43:29.337032 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-c2e0bd85-5c23-4f82-8af0-8fc9618ca287 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:43:29.383781 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:43:29.397842 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1390/2788] 10.222.0.50 () {58 vars in 958 bytes} [Tue Sep 13 16:43:29 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:43:29.405471 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8425ceeb-cfff-4345-b38c-7d176c6bb092 None None] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:43:29.405760 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8425ceeb-cfff-4345-b38c-7d176c6bb092 None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:43:29.406038 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8425ceeb-cfff-4345-b38c-7d176c6bb092 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:43:29.425757 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.auth.core [None req-8425ceeb-cfff-4345-b38c-7d176c6bb092 None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81437) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:43:29.426994 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-8425ceeb-cfff-4345-b38c-7d176c6bb092 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:43:29.434812 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1399/2789] 10.222.0.50 () {58 vars in 958 bytes} [Tue Sep 13 16:43:29 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:43:29.438988 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2e0bd85-5c23-4f82-8af0-8fc9618ca287 None None] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:43:29.449534 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2e0bd85-5c23-4f82-8af0-8fc9618ca287 None None] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:43:29.454575 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51742aee-9bbf-4a8d-9577-4833ce18856f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:43:29.455228 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51742aee-9bbf-4a8d-9577-4833ce18856f admin admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:43:29.456284 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51742aee-9bbf-4a8d-9577-4833ce18856f admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:43:29.456570 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51742aee-9bbf-4a8d-9577-4833ce18856f admin admin] PATH_INFO: `/v3/domains` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:43:29.457312 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51742aee-9bbf-4a8d-9577-4833ce18856f admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81438) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 16:43:29.457616 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51742aee-9bbf-4a8d-9577-4833ce18856f admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:43:29.459347 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51742aee-9bbf-4a8d-9577-4833ce18856f admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:43:29.470757 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1391/2790] 10.222.0.50 () {58 vars in 1153 bytes} [Tue Sep 13 16:43:29 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 16:43:29.475429 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8425ceeb-cfff-4345-b38c-7d176c6bb092 None None] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:43:29.484722 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8425ceeb-cfff-4345-b38c-7d176c6bb092 None None] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:43:29.489412 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5764131a-d501-4fc4-8f8c-7f0a47d7e2ef admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:43:29.490068 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5764131a-d501-4fc4-8f8c-7f0a47d7e2ef admin admin] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:43:29.490346 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5764131a-d501-4fc4-8f8c-7f0a47d7e2ef admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:43:29.490630 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5764131a-d501-4fc4-8f8c-7f0a47d7e2ef admin admin] PATH_INFO: `/v3/projects` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:43:29.491313 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5764131a-d501-4fc4-8f8c-7f0a47d7e2ef admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:43:29.493222 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5764131a-d501-4fc4-8f8c-7f0a47d7e2ef admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:43:29.520929 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1400/2791] 10.222.0.50 () {60 vars in 1153 bytes} [Tue Sep 13 16:43:29 2022] POST /identity/v3/projects => generated 391 bytes in 46 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 16:43:29.525017 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51742aee-9bbf-4a8d-9577-4833ce18856f admin admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:43:29.535562 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51742aee-9bbf-4a8d-9577-4833ce18856f admin admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:43:29.592198 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-642a73c4-23e0-4088-8070-61f32465bf38 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:43:29.592795 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-642a73c4-23e0-4088-8070-61f32465bf38 admin admin] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:43:29.593071 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-642a73c4-23e0-4088-8070-61f32465bf38 admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:43:29.593340 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-642a73c4-23e0-4088-8070-61f32465bf38 admin admin] PATH_INFO: `/v3/users` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:43:29.593955 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-642a73c4-23e0-4088-8070-61f32465bf38 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:43:29.595917 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-642a73c4-23e0-4088-8070-61f32465bf38 admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:43:29.618077 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1392/2792] 10.222.0.50 () {60 vars in 1144 bytes} [Tue Sep 13 16:43:29 2022] POST /identity/v3/users => generated 296 bytes in 93 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 16:43:29.622181 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5764131a-d501-4fc4-8f8c-7f0a47d7e2ef admin admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:43:29.633174 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5764131a-d501-4fc4-8f8c-7f0a47d7e2ef admin admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:43:29.637807 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2caf7eb5-d41e-48bc-9172-4816ac617444 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:43:29.638404 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2caf7eb5-d41e-48bc-9172-4816ac617444 admin admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:43:29.638683 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2caf7eb5-d41e-48bc-9172-4816ac617444 admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:43:29.638984 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2caf7eb5-d41e-48bc-9172-4816ac617444 admin admin] PATH_INFO: `/v3/roles` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:43:29.639624 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2caf7eb5-d41e-48bc-9172-4816ac617444 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:43:29.641213 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2caf7eb5-d41e-48bc-9172-4816ac617444 admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:43:29.651883 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1401/2793] 10.222.0.50 () {58 vars in 1122 bytes} [Tue Sep 13 16:43:29 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 16:43:29.655683 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-642a73c4-23e0-4088-8070-61f32465bf38 admin admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:43:29.666355 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-642a73c4-23e0-4088-8070-61f32465bf38 admin admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:43:29.671310 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-981540d4-a13d-451e-9789-46340c40359f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:43:29.671903 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-981540d4-a13d-451e-9789-46340c40359f admin admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:43:29.672182 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-981540d4-a13d-451e-9789-46340c40359f admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:43:29.672457 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-981540d4-a13d-451e-9789-46340c40359f admin admin] PATH_INFO: `/v3/roles` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:43:29.673093 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-981540d4-a13d-451e-9789-46340c40359f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:43:29.675422 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-981540d4-a13d-451e-9789-46340c40359f admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:43:29.685930 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1393/2794] 10.222.0.50 () {58 vars in 1122 bytes} [Tue Sep 13 16:43:29 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 13 16:43:29.690574 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2caf7eb5-d41e-48bc-9172-4816ac617444 admin admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:43:29.700266 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2caf7eb5-d41e-48bc-9172-4816ac617444 admin admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:43:29.704875 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a755bf54-1c2f-43b9-8f70-b99548b2881b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:43:29.705897 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a755bf54-1c2f-43b9-8f70-b99548b2881b admin admin] REQUEST_METHOD: `PUT` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:43:29.706173 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a755bf54-1c2f-43b9-8f70-b99548b2881b admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:43:29.706448 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a755bf54-1c2f-43b9-8f70-b99548b2881b admin admin] PATH_INFO: `/v3/projects/145165e302bd4a238e1d851ae98edffe/users/4e5b369f0d794b39ab317aed56f751cb/roles/15db4aaca3264ae29bde38903ba43471` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:43:29.720662 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a755bf54-1c2f-43b9-8f70-b99548b2881b admin admin] RBAC: Authorizing `identity:create_grant(project_id=145165e302bd4a238e1d851ae98edffe, user_id=4e5b369f0d794b39ab317aed56f751cb, role_id=15db4aaca3264ae29bde38903ba43471)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:43:29.723463 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a755bf54-1c2f-43b9-8f70-b99548b2881b admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:43:29.735744 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1402/2795] 10.222.0.50 () {60 vars in 1483 bytes} [Tue Sep 13 16:43:29 2022] PUT /identity/v3/projects/145165e302bd4a238e1d851ae98edffe/users/4e5b369f0d794b39ab317aed56f751cb/roles/15db4aaca3264ae29bde38903ba43471 => generated 0 bytes in 46 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 16:43:29.746405 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f369edb1-02a7-4c86-871c-4ac2ff990c02 None None] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:43:29.746695 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f369edb1-02a7-4c86-871c-4ac2ff990c02 None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:43:29.747006 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f369edb1-02a7-4c86-871c-4ac2ff990c02 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:43:29.776270 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.auth.core [None req-f369edb1-02a7-4c86-871c-4ac2ff990c02 None None] MFA Rules not processed for user `4e5b369f0d794b39ab317aed56f751cb`. Rule list: `[]` (Enabled: `True`). {{(pid=81438) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:43:29.777476 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-f369edb1-02a7-4c86-871c-4ac2ff990c02 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:43:29.842337 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1394/2796] 10.222.0.50 () {58 vars in 958 bytes} [Tue Sep 13 16:43:29 2022] POST /identity/v3/auth/tokens => generated 3268 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:43:29.853096 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a755bf54-1c2f-43b9-8f70-b99548b2881b admin admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:43:29.854805 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-a755bf54-1c2f-43b9-8f70-b99548b2881b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:43:29.860189 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:43:29.914286 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a755bf54-1c2f-43b9-8f70-b99548b2881b admin admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:43:29.918081 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa9e8b7e-12e1-4e45-8d43-9b69d4026561 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7fd6b5afd67b48f9b0f0597ccd4cf4df', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:43:29.918682 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa9e8b7e-12e1-4e45-8d43-9b69d4026561 service cinder] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:43:29.918998 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa9e8b7e-12e1-4e45-8d43-9b69d4026561 service cinder] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:43:29.919272 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa9e8b7e-12e1-4e45-8d43-9b69d4026561 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:43:29.928468 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa9e8b7e-12e1-4e45-8d43-9b69d4026561 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:43:29.929898 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa9e8b7e-12e1-4e45-8d43-9b69d4026561 service cinder] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:43:29.938833 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1403/2797] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 16:43:29 2022] GET /identity/v3/auth/tokens => generated 3268 bytes in 86 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 16:43:37.651976 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f369edb1-02a7-4c86-871c-4ac2ff990c02 None None] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:43:37.653320 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-f369edb1-02a7-4c86-871c-4ac2ff990c02 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:43:37.706308 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:43:37.710495 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f369edb1-02a7-4c86-871c-4ac2ff990c02 None None] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:43:37.715197 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c97c3df-eba2-43fb-b4ee-2633ea582c25 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f436f07939d4855a21b195bc7566244', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:43:37.715811 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c97c3df-eba2-43fb-b4ee-2633ea582c25 service neutron] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:43:37.716091 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c97c3df-eba2-43fb-b4ee-2633ea582c25 service neutron] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:43:37.716364 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c97c3df-eba2-43fb-b4ee-2633ea582c25 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:43:37.718092 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-9c97c3df-eba2-43fb-b4ee-2633ea582c25 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:43:37.728404 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c97c3df-eba2-43fb-b4ee-2633ea582c25 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:43:37.729818 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c97c3df-eba2-43fb-b4ee-2633ea582c25 service neutron] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:43:37.738407 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1395/2798] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 16:43:37 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 87 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 16:43:39.934654 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=25 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:43:47.734118 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=4 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:43:48.179316 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa9e8b7e-12e1-4e45-8d43-9b69d4026561 service cinder] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:43:48.242128 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa9e8b7e-12e1-4e45-8d43-9b69d4026561 service cinder] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:43:48.246653 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14462569-3058-4fa2-84da-a458481b45c3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:43:48.247344 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14462569-3058-4fa2-84da-a458481b45c3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:43:48.247736 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14462569-3058-4fa2-84da-a458481b45c3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:43:48.248017 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14462569-3058-4fa2-84da-a458481b45c3 admin admin] PATH_INFO: `/v3/users/4e5b369f0d794b39ab317aed56f751cb` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:43:48.249141 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14462569-3058-4fa2-84da-a458481b45c3 admin admin] RBAC: Authorizing `identity:delete_user(user_id=4e5b369f0d794b39ab317aed56f751cb)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:43:48.251106 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14462569-3058-4fa2-84da-a458481b45c3 admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:43:48.299114 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-14462569-3058-4fa2-84da-a458481b45c3 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '4e5b369f0d794b39ab317aed56f751cb'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:43:48.306351 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-14462569-3058-4fa2-84da-a458481b45c3 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '4e5b369f0d794b39ab317aed56f751cb'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:43:48.318854 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-14462569-3058-4fa2-84da-a458481b45c3 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '4e5b369f0d794b39ab317aed56f751cb'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:43:48.319536 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-14462569-3058-4fa2-84da-a458481b45c3 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '4e5b369f0d794b39ab317aed56f751cb'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:43:48.326416 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-14462569-3058-4fa2-84da-a458481b45c3 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '4e5b369f0d794b39ab317aed56f751cb'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:43:48.332709 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1404/2799] 10.222.0.50 () {58 vars in 1224 bytes} [Tue Sep 13 16:43:48 2022] DELETE /identity/v3/users/4e5b369f0d794b39ab317aed56f751cb => generated 0 bytes in 154 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 16:43:48.337132 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15335b0b-3b70-4550-9575-83030e1bdab9 None None] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:43:48.337425 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15335b0b-3b70-4550-9575-83030e1bdab9 None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:43:48.337705 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15335b0b-3b70-4550-9575-83030e1bdab9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:43:48.357125 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.auth.core [None req-15335b0b-3b70-4550-9575-83030e1bdab9 None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81438) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:43:48.358541 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-15335b0b-3b70-4550-9575-83030e1bdab9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:43:48.418301 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1396/2800] 10.222.0.50 () {58 vars in 958 bytes} [Tue Sep 13 16:43:48 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 82 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:43:48.427931 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14462569-3058-4fa2-84da-a458481b45c3 admin admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:43:48.429332 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-14462569-3058-4fa2-84da-a458481b45c3 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:43:48.486838 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14462569-3058-4fa2-84da-a458481b45c3 admin admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:43:48.490838 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-404a6884-c078-4a38-9446-35559d10ff14 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f436f07939d4855a21b195bc7566244', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:43:48.491468 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-404a6884-c078-4a38-9446-35559d10ff14 service neutron] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:43:48.491746 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-404a6884-c078-4a38-9446-35559d10ff14 service neutron] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:43:48.492022 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-404a6884-c078-4a38-9446-35559d10ff14 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:43:48.500945 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-404a6884-c078-4a38-9446-35559d10ff14 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:43:48.502406 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-404a6884-c078-4a38-9446-35559d10ff14 service neutron] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:43:48.511780 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1405/2801] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 16:43:48 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 16:43:48.790962 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15335b0b-3b70-4550-9575-83030e1bdab9 None None] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:43:48.792544 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-15335b0b-3b70-4550-9575-83030e1bdab9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:43:48.805871 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15335b0b-3b70-4550-9575-83030e1bdab9 None None] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:43:48.810577 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0700a09-5a1e-43cb-a0c1-1126c802bda8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:43:48.811247 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0700a09-5a1e-43cb-a0c1-1126c802bda8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:43:48.811528 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0700a09-5a1e-43cb-a0c1-1126c802bda8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:43:48.811803 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0700a09-5a1e-43cb-a0c1-1126c802bda8 admin admin] PATH_INFO: `/v3/projects/145165e302bd4a238e1d851ae98edffe` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:43:48.813018 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0700a09-5a1e-43cb-a0c1-1126c802bda8 admin admin] RBAC: Authorizing `identity:delete_project(project_id=145165e302bd4a238e1d851ae98edffe)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:43:48.815545 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0700a09-5a1e-43cb-a0c1-1126c802bda8 admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:43:48.863697 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-c0700a09-5a1e-43cb-a0c1-1126c802bda8 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '145165e302bd4a238e1d851ae98edffe', 'request_id': 'req-c0700a09-5a1e-43cb-a0c1-1126c802bda8'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:43:48.870726 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-c0700a09-5a1e-43cb-a0c1-1126c802bda8 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '145165e302bd4a238e1d851ae98edffe', 'request_id': 'req-c0700a09-5a1e-43cb-a0c1-1126c802bda8'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:43:48.876324 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-c0700a09-5a1e-43cb-a0c1-1126c802bda8 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '145165e302bd4a238e1d851ae98edffe', 'request_id': 'req-c0700a09-5a1e-43cb-a0c1-1126c802bda8'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:43:48.888145 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-c0700a09-5a1e-43cb-a0c1-1126c802bda8 admin admin] The token cache is being invalidate because project 145165e302bd4a238e1d851ae98edffe was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81438) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 13 16:43:48.888433 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-c0700a09-5a1e-43cb-a0c1-1126c802bda8 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:43:48.889066 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-c0700a09-5a1e-43cb-a0c1-1126c802bda8 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:43:48.890528 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1397/2802] 10.222.0.50 () {58 vars in 1233 bytes} [Tue Sep 13 16:43:48 2022] DELETE /identity/v3/projects/145165e302bd4a238e1d851ae98edffe => generated 0 bytes in 100 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 16:43:48.896701 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bdef8ae-c9ad-44a0-bc18-86ca0dbbce2e None None] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:43:48.897020 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bdef8ae-c9ad-44a0-bc18-86ca0dbbce2e None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:43:48.897297 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bdef8ae-c9ad-44a0-bc18-86ca0dbbce2e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:43:48.916596 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.auth.core [None req-0bdef8ae-c9ad-44a0-bc18-86ca0dbbce2e None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81437) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:43:48.917796 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-0bdef8ae-c9ad-44a0-bc18-86ca0dbbce2e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:43:48.936145 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:43:48.982194 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1406/2803] 10.222.0.50 () {58 vars in 958 bytes} [Tue Sep 13 16:43:48 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:43:48.990872 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-427bd41e-697c-4fc5-962b-46a5969ad3f2 None None] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:43:48.991192 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-427bd41e-697c-4fc5-962b-46a5969ad3f2 None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:43:48.991473 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-427bd41e-697c-4fc5-962b-46a5969ad3f2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:43:49.013412 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.auth.core [None req-427bd41e-697c-4fc5-962b-46a5969ad3f2 None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81438) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:43:49.014630 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-427bd41e-697c-4fc5-962b-46a5969ad3f2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:43:49.023718 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1398/2804] 10.222.0.50 () {58 vars in 958 bytes} [Tue Sep 13 16:43:48 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:43:49.027907 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bdef8ae-c9ad-44a0-bc18-86ca0dbbce2e None None] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:43:49.037823 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bdef8ae-c9ad-44a0-bc18-86ca0dbbce2e None None] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:43:49.042462 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5624186f-dda9-47f9-b07c-668f574da142 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:43:49.043100 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5624186f-dda9-47f9-b07c-668f574da142 admin admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:43:49.043412 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5624186f-dda9-47f9-b07c-668f574da142 admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:43:49.043683 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5624186f-dda9-47f9-b07c-668f574da142 admin admin] PATH_INFO: `/v3/domains` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:43:49.044322 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5624186f-dda9-47f9-b07c-668f574da142 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81437) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 16:43:49.044654 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5624186f-dda9-47f9-b07c-668f574da142 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:43:49.046372 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5624186f-dda9-47f9-b07c-668f574da142 admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:43:49.061163 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1407/2805] 10.222.0.50 () {58 vars in 1153 bytes} [Tue Sep 13 16:43:49 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 16:43:49.062411 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-427bd41e-697c-4fc5-962b-46a5969ad3f2 None None] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:43:49.072220 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-427bd41e-697c-4fc5-962b-46a5969ad3f2 None None] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:43:49.076841 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43b82226-0feb-4b0c-a0a6-0b5877f9c995 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:43:49.077460 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43b82226-0feb-4b0c-a0a6-0b5877f9c995 admin admin] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:43:49.077739 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43b82226-0feb-4b0c-a0a6-0b5877f9c995 admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:43:49.078010 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43b82226-0feb-4b0c-a0a6-0b5877f9c995 admin admin] PATH_INFO: `/v3/projects` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:43:49.078627 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43b82226-0feb-4b0c-a0a6-0b5877f9c995 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:43:49.081517 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43b82226-0feb-4b0c-a0a6-0b5877f9c995 admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:43:49.108572 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1399/2806] 10.222.0.50 () {60 vars in 1153 bytes} [Tue Sep 13 16:43:49 2022] POST /identity/v3/projects => generated 385 bytes in 47 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 16:43:49.112887 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5624186f-dda9-47f9-b07c-668f574da142 admin admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:43:49.122638 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5624186f-dda9-47f9-b07c-668f574da142 admin admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:43:49.180325 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c33562d6-7df1-4e5f-ad07-edca7ab80a1c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:43:49.180973 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c33562d6-7df1-4e5f-ad07-edca7ab80a1c admin admin] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:43:49.181299 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c33562d6-7df1-4e5f-ad07-edca7ab80a1c admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:43:49.181574 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c33562d6-7df1-4e5f-ad07-edca7ab80a1c admin admin] PATH_INFO: `/v3/users` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:43:49.182239 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c33562d6-7df1-4e5f-ad07-edca7ab80a1c admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:43:49.184408 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c33562d6-7df1-4e5f-ad07-edca7ab80a1c admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:43:49.204988 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1408/2807] 10.222.0.50 () {60 vars in 1144 bytes} [Tue Sep 13 16:43:49 2022] POST /identity/v3/users => generated 293 bytes in 92 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 16:43:49.209323 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43b82226-0feb-4b0c-a0a6-0b5877f9c995 admin admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:43:49.219295 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43b82226-0feb-4b0c-a0a6-0b5877f9c995 admin admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:43:49.224026 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4989b269-dc18-4522-876e-e20405b371b6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:43:49.224618 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4989b269-dc18-4522-876e-e20405b371b6 admin admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:43:49.224896 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4989b269-dc18-4522-876e-e20405b371b6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:43:49.225166 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4989b269-dc18-4522-876e-e20405b371b6 admin admin] PATH_INFO: `/v3/roles` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:43:49.225796 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4989b269-dc18-4522-876e-e20405b371b6 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:43:49.227435 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4989b269-dc18-4522-876e-e20405b371b6 admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:43:49.237654 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1400/2808] 10.222.0.50 () {58 vars in 1122 bytes} [Tue Sep 13 16:43:49 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 16:43:49.241513 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c33562d6-7df1-4e5f-ad07-edca7ab80a1c admin admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:43:49.251358 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c33562d6-7df1-4e5f-ad07-edca7ab80a1c admin admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:43:49.256559 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1419f9f-ac5e-46c4-8826-9580580a5648 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:43:49.257153 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1419f9f-ac5e-46c4-8826-9580580a5648 admin admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:43:49.257432 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1419f9f-ac5e-46c4-8826-9580580a5648 admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:43:49.257704 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1419f9f-ac5e-46c4-8826-9580580a5648 admin admin] PATH_INFO: `/v3/roles` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:43:49.258341 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1419f9f-ac5e-46c4-8826-9580580a5648 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:43:49.259969 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1419f9f-ac5e-46c4-8826-9580580a5648 admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:43:49.271094 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1409/2809] 10.222.0.50 () {58 vars in 1122 bytes} [Tue Sep 13 16:43:49 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 16:43:49.274980 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4989b269-dc18-4522-876e-e20405b371b6 admin admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:43:49.284719 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4989b269-dc18-4522-876e-e20405b371b6 admin admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:43:49.289439 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5307183a-683b-454c-bd33-a87342448485 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:43:49.290112 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5307183a-683b-454c-bd33-a87342448485 admin admin] REQUEST_METHOD: `PUT` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:43:49.290392 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5307183a-683b-454c-bd33-a87342448485 admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:43:49.290664 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5307183a-683b-454c-bd33-a87342448485 admin admin] PATH_INFO: `/v3/projects/08f6037f42c34be7b2407c258d8885be/users/91ce9cad1eab41efb4da9503c23ad073/roles/15db4aaca3264ae29bde38903ba43471` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:43:49.305501 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5307183a-683b-454c-bd33-a87342448485 admin admin] RBAC: Authorizing `identity:create_grant(project_id=08f6037f42c34be7b2407c258d8885be, user_id=91ce9cad1eab41efb4da9503c23ad073, role_id=15db4aaca3264ae29bde38903ba43471)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:43:49.308155 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5307183a-683b-454c-bd33-a87342448485 admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:43:49.320651 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1401/2810] 10.222.0.50 () {60 vars in 1483 bytes} [Tue Sep 13 16:43:49 2022] PUT /identity/v3/projects/08f6037f42c34be7b2407c258d8885be/users/91ce9cad1eab41efb4da9503c23ad073/roles/15db4aaca3264ae29bde38903ba43471 => generated 0 bytes in 46 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 16:43:49.325504 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d80385a7-8e54-49a9-9d5b-3a4c30ae5e1d None None] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:43:49.325792 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d80385a7-8e54-49a9-9d5b-3a4c30ae5e1d None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:43:49.326072 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d80385a7-8e54-49a9-9d5b-3a4c30ae5e1d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:43:49.345058 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.auth.core [None req-d80385a7-8e54-49a9-9d5b-3a4c30ae5e1d None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81437) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:43:49.346284 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-d80385a7-8e54-49a9-9d5b-3a4c30ae5e1d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:43:49.405295 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1410/2811] 10.222.0.50 () {58 vars in 958 bytes} [Tue Sep 13 16:43:49 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 81 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:43:49.415108 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5307183a-683b-454c-bd33-a87342448485 admin admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:43:49.416808 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-5307183a-683b-454c-bd33-a87342448485 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:43:49.438488 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:43:49.476420 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5307183a-683b-454c-bd33-a87342448485 admin admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:43:49.479977 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1aef0604-cfb6-44be-bb20-8ee5b95a908a service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f436f07939d4855a21b195bc7566244', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:43:49.480572 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1aef0604-cfb6-44be-bb20-8ee5b95a908a service neutron] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:43:49.480848 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1aef0604-cfb6-44be-bb20-8ee5b95a908a service neutron] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:43:49.481128 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1aef0604-cfb6-44be-bb20-8ee5b95a908a service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:43:49.489867 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1aef0604-cfb6-44be-bb20-8ee5b95a908a service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:43:49.491316 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1aef0604-cfb6-44be-bb20-8ee5b95a908a service neutron] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:43:49.500398 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1402/2812] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 16:43:49 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 16:43:53.454178 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49117290-52a0-490f-abe2-7d13c67f100b None None] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:43:53.455529 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49117290-52a0-490f-abe2-7d13c67f100b None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:43:53.455648 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49117290-52a0-490f-abe2-7d13c67f100b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:43:53.493260 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.auth.core [None req-49117290-52a0-490f-abe2-7d13c67f100b None None] MFA Rules not processed for user `91ce9cad1eab41efb4da9503c23ad073`. Rule list: `[]` (Enabled: `True`). {{(pid=81437) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:43:53.494513 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-49117290-52a0-490f-abe2-7d13c67f100b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:43:53.519857 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:43:53.570224 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1411/2813] 10.222.0.50 () {58 vars in 958 bytes} [Tue Sep 13 16:43:53 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 117 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:43:53.574717 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1aef0604-cfb6-44be-bb20-8ee5b95a908a service neutron] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:43:53.585204 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1aef0604-cfb6-44be-bb20-8ee5b95a908a service neutron] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:43:53.589671 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eab45ad8-901f-4461-bc5d-8a2e62cf2ad1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:43:53.590271 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eab45ad8-901f-4461-bc5d-8a2e62cf2ad1 admin admin] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:43:53.590547 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eab45ad8-901f-4461-bc5d-8a2e62cf2ad1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:43:53.590814 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eab45ad8-901f-4461-bc5d-8a2e62cf2ad1 admin admin] PATH_INFO: `/v3/projects` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:43:53.591476 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eab45ad8-901f-4461-bc5d-8a2e62cf2ad1 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:43:53.593432 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eab45ad8-901f-4461-bc5d-8a2e62cf2ad1 admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:43:53.621976 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1403/2814] 10.222.0.50 () {60 vars in 1153 bytes} [Tue Sep 13 16:43:53 2022] POST /identity/v3/projects => generated 383 bytes in 48 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 16:43:53.627992 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49117290-52a0-490f-abe2-7d13c67f100b None None] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:43:53.653319 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49117290-52a0-490f-abe2-7d13c67f100b None None] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:43:53.719951 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-799d3233-f0c1-4b10-aee5-e28270c5c54e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:43:53.721137 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-799d3233-f0c1-4b10-aee5-e28270c5c54e admin admin] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:43:53.721415 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-799d3233-f0c1-4b10-aee5-e28270c5c54e admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:43:53.721686 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-799d3233-f0c1-4b10-aee5-e28270c5c54e admin admin] PATH_INFO: `/v3/users` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:43:53.722326 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-799d3233-f0c1-4b10-aee5-e28270c5c54e admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:43:53.724301 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-799d3233-f0c1-4b10-aee5-e28270c5c54e admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:43:53.749298 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1412/2815] 10.222.0.50 () {60 vars in 1144 bytes} [Tue Sep 13 16:43:53 2022] POST /identity/v3/users => generated 298 bytes in 122 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 16:43:53.753323 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eab45ad8-901f-4461-bc5d-8a2e62cf2ad1 admin admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:43:53.764129 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eab45ad8-901f-4461-bc5d-8a2e62cf2ad1 admin admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:43:53.775853 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f36c4480-d714-450c-b9b0-42b1e9774ef4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:43:53.776486 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f36c4480-d714-450c-b9b0-42b1e9774ef4 admin admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:43:53.776822 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f36c4480-d714-450c-b9b0-42b1e9774ef4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:43:53.777095 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f36c4480-d714-450c-b9b0-42b1e9774ef4 admin admin] PATH_INFO: `/v3/roles` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:43:53.777741 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f36c4480-d714-450c-b9b0-42b1e9774ef4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:43:53.779381 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f36c4480-d714-450c-b9b0-42b1e9774ef4 admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:43:53.791531 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1404/2816] 10.222.0.50 () {58 vars in 1122 bytes} [Tue Sep 13 16:43:53 2022] GET /identity/v3/roles => generated 1420 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 16:43:53.795870 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-799d3233-f0c1-4b10-aee5-e28270c5c54e admin admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:43:53.809424 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-799d3233-f0c1-4b10-aee5-e28270c5c54e admin admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:43:53.815158 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f3d75e5-6164-4797-bfa8-6d2e69134085 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:43:53.815908 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f3d75e5-6164-4797-bfa8-6d2e69134085 admin admin] REQUEST_METHOD: `PUT` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:43:53.816188 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f3d75e5-6164-4797-bfa8-6d2e69134085 admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:43:53.816462 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f3d75e5-6164-4797-bfa8-6d2e69134085 admin admin] PATH_INFO: `/v3/projects/a8cb6e56acdb45c6aafdd3024471e8eb/users/2d30f26e52424a7e92a0099fbc480683/roles/031d4e1a23e34f71858fc4a6dfbd4ccb` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:43:53.833341 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f3d75e5-6164-4797-bfa8-6d2e69134085 admin admin] RBAC: Authorizing `identity:create_grant(project_id=a8cb6e56acdb45c6aafdd3024471e8eb, user_id=2d30f26e52424a7e92a0099fbc480683, role_id=031d4e1a23e34f71858fc4a6dfbd4ccb)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:43:53.836038 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f3d75e5-6164-4797-bfa8-6d2e69134085 admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:43:53.850679 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1413/2817] 10.222.0.50 () {60 vars in 1483 bytes} [Tue Sep 13 16:43:53 2022] PUT /identity/v3/projects/a8cb6e56acdb45c6aafdd3024471e8eb/users/2d30f26e52424a7e92a0099fbc480683/roles/031d4e1a23e34f71858fc4a6dfbd4ccb => generated 0 bytes in 55 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 16:43:53.854581 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f36c4480-d714-450c-b9b0-42b1e9774ef4 admin admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:43:53.865557 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f36c4480-d714-450c-b9b0-42b1e9774ef4 admin admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:43:53.929209 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc4f11b4-852d-4e70-8a55-d4dfea28a1e4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:43:53.930367 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc4f11b4-852d-4e70-8a55-d4dfea28a1e4 admin admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:43:53.931097 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc4f11b4-852d-4e70-8a55-d4dfea28a1e4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:43:53.931696 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc4f11b4-852d-4e70-8a55-d4dfea28a1e4 admin admin] PATH_INFO: `/v3/roles` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:43:53.933018 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc4f11b4-852d-4e70-8a55-d4dfea28a1e4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:43:53.934990 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc4f11b4-852d-4e70-8a55-d4dfea28a1e4 admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:43:53.948168 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1405/2818] 10.222.0.50 () {58 vars in 1122 bytes} [Tue Sep 13 16:43:53 2022] GET /identity/v3/roles => generated 1420 bytes in 94 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 16:43:53.952425 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f3d75e5-6164-4797-bfa8-6d2e69134085 admin admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:43:53.965741 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f3d75e5-6164-4797-bfa8-6d2e69134085 admin admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:43:53.970815 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0abcb82-4697-469b-b6b7-3fe917fd919e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:43:53.971637 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0abcb82-4697-469b-b6b7-3fe917fd919e admin admin] REQUEST_METHOD: `PUT` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:43:53.971917 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0abcb82-4697-469b-b6b7-3fe917fd919e admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:43:53.972225 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0abcb82-4697-469b-b6b7-3fe917fd919e admin admin] PATH_INFO: `/v3/projects/a8cb6e56acdb45c6aafdd3024471e8eb/users/2d30f26e52424a7e92a0099fbc480683/roles/031d4e1a23e34f71858fc4a6dfbd4ccb` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:43:53.974174 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0abcb82-4697-469b-b6b7-3fe917fd919e admin admin] RBAC: Authorizing `identity:create_grant(project_id=a8cb6e56acdb45c6aafdd3024471e8eb, user_id=2d30f26e52424a7e92a0099fbc480683, role_id=031d4e1a23e34f71858fc4a6dfbd4ccb)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:43:53.976897 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0abcb82-4697-469b-b6b7-3fe917fd919e admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:43:53.989879 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1414/2819] 10.222.0.50 () {60 vars in 1483 bytes} [Tue Sep 13 16:43:53 2022] PUT /identity/v3/projects/a8cb6e56acdb45c6aafdd3024471e8eb/users/2d30f26e52424a7e92a0099fbc480683/roles/031d4e1a23e34f71858fc4a6dfbd4ccb => generated 0 bytes in 38 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 13 16:43:58.581897 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da7bc33c-28b4-49a9-8884-df36f0164568 None None] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:43:58.581897 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da7bc33c-28b4-49a9-8884-df36f0164568 None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:43:58.581897 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da7bc33c-28b4-49a9-8884-df36f0164568 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:43:58.627479 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.auth.core [None req-da7bc33c-28b4-49a9-8884-df36f0164568 None None] MFA Rules not processed for user `2d30f26e52424a7e92a0099fbc480683`. Rule list: `[]` (Enabled: `True`). {{(pid=81438) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:43:58.628786 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-da7bc33c-28b4-49a9-8884-df36f0164568 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:43:58.745652 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1406/2820] 10.222.0.50 () {58 vars in 958 bytes} [Tue Sep 13 16:43:58 2022] POST /identity/v3/auth/tokens => generated 3327 bytes in 170 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:43:58.760389 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0abcb82-4697-469b-b6b7-3fe917fd919e admin admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:43:58.762338 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-d0abcb82-4697-469b-b6b7-3fe917fd919e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:43:58.828970 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0abcb82-4697-469b-b6b7-3fe917fd919e admin admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:43:58.833598 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d98128a7-7249-4922-aa20-2484b3ae2b55 service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6ab5f8018f464be69d4af7683f553738', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:43:58.834157 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d98128a7-7249-4922-aa20-2484b3ae2b55 service glance] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:43:58.834439 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d98128a7-7249-4922-aa20-2484b3ae2b55 service glance] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:43:58.834719 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d98128a7-7249-4922-aa20-2484b3ae2b55 service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:43:58.861178 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:43:58.896870 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d98128a7-7249-4922-aa20-2484b3ae2b55 service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:43:58.898321 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d98128a7-7249-4922-aa20-2484b3ae2b55 service glance] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:43:58.907910 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1415/2821] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 16:43:58 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 148 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 16:43:59.047722 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69a991b5-85b3-485c-b21b-248fa0695b4f None None] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:43:59.048549 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69a991b5-85b3-485c-b21b-248fa0695b4f None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:43:59.048902 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69a991b5-85b3-485c-b21b-248fa0695b4f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:43:59.068621 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.auth.core [None req-69a991b5-85b3-485c-b21b-248fa0695b4f None None] MFA Rules not processed for user `f8364b1a190d4218bf353a89bdb81448`. Rule list: `[]` (Enabled: `True`). {{(pid=81438) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:43:59.069855 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-69a991b5-85b3-485c-b21b-248fa0695b4f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:43:59.075323 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:43:59.127163 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1407/2822] 10.222.0.50 () {58 vars in 1009 bytes} [Tue Sep 13 16:43:59 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 80 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:43:59.139289 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d98128a7-7249-4922-aa20-2484b3ae2b55 service glance] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:43:59.140730 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-d98128a7-7249-4922-aa20-2484b3ae2b55 service glance] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:43:59.211793 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d98128a7-7249-4922-aa20-2484b3ae2b55 service glance] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:43:59.216791 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a0d16dc-1288-4367-98c9-70833ebf6b8d service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b00ab0d6321d483188bc69deaddcdbbc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:43:59.217403 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a0d16dc-1288-4367-98c9-70833ebf6b8d service swift] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:43:59.217681 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a0d16dc-1288-4367-98c9-70833ebf6b8d service swift] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:43:59.217955 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a0d16dc-1288-4367-98c9-70833ebf6b8d service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:43:59.226041 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a0d16dc-1288-4367-98c9-70833ebf6b8d service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:43:59.227659 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a0d16dc-1288-4367-98c9-70833ebf6b8d service swift] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:43:59.236070 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1416/2823] 10.222.0.50 () {60 vars in 1384 bytes} [Tue Sep 13 16:43:59 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 97 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Sep 13 16:44:08.047160 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc54ced3-fd83-479b-89e5-4491d77424a8 None None] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:44:08.047720 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc54ced3-fd83-479b-89e5-4491d77424a8 None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:44:08.047720 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc54ced3-fd83-479b-89e5-4491d77424a8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:44:08.067569 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.auth.core [None req-bc54ced3-fd83-479b-89e5-4491d77424a8 None None] MFA Rules not processed for user `91ce9cad1eab41efb4da9503c23ad073`. Rule list: `[]` (Enabled: `True`). {{(pid=81438) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:44:08.068800 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-bc54ced3-fd83-479b-89e5-4491d77424a8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:44:08.075664 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1408/2824] 10.222.0.50 () {58 vars in 958 bytes} [Tue Sep 13 16:44:08 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:44:09.232341 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=48 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:44:09.862008 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51fcbdac-96cb-4413-b7f5-87b92b0a4bd4 None None] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:44:09.862289 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51fcbdac-96cb-4413-b7f5-87b92b0a4bd4 None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:44:09.862563 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51fcbdac-96cb-4413-b7f5-87b92b0a4bd4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:44:09.881683 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.auth.core [None req-51fcbdac-96cb-4413-b7f5-87b92b0a4bd4 None None] MFA Rules not processed for user `f8364b1a190d4218bf353a89bdb81448`. Rule list: `[]` (Enabled: `True`). {{(pid=81437) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:44:09.882905 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-51fcbdac-96cb-4413-b7f5-87b92b0a4bd4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:44:09.890130 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1417/2825] 10.222.0.50 () {58 vars in 1009 bytes} [Tue Sep 13 16:44:09 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:44:09.902930 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc54ced3-fd83-479b-89e5-4491d77424a8 None None] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:44:09.920216 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc54ced3-fd83-479b-89e5-4491d77424a8 None None] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:44:09.926258 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afe4689f-060d-45f2-b6f9-ae25b0336dff service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b00ab0d6321d483188bc69deaddcdbbc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:44:09.926863 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afe4689f-060d-45f2-b6f9-ae25b0336dff service swift] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:44:09.927341 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afe4689f-060d-45f2-b6f9-ae25b0336dff service swift] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:44:09.927609 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afe4689f-060d-45f2-b6f9-ae25b0336dff service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:44:09.936226 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afe4689f-060d-45f2-b6f9-ae25b0336dff service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:44:09.938031 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afe4689f-060d-45f2-b6f9-ae25b0336dff service swift] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:44:09.946882 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1409/2826] 10.222.0.50 () {60 vars in 1384 bytes} [Tue Sep 13 16:44:09 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 44 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Sep 13 16:44:19.876443 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=3 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:44:19.943854 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=29 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:44:35.693098 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51fcbdac-96cb-4413-b7f5-87b92b0a4bd4 None None] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:44:35.694512 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-51fcbdac-96cb-4413-b7f5-87b92b0a4bd4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:44:35.749340 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51fcbdac-96cb-4413-b7f5-87b92b0a4bd4 None None] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:44:35.753492 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bcd5d44-5e15-49a5-930f-b446e95e2a5d service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f436f07939d4855a21b195bc7566244', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:44:35.754097 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bcd5d44-5e15-49a5-930f-b446e95e2a5d service neutron] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:44:35.754372 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bcd5d44-5e15-49a5-930f-b446e95e2a5d service neutron] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:44:35.754644 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bcd5d44-5e15-49a5-930f-b446e95e2a5d service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:44:35.763428 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0bcd5d44-5e15-49a5-930f-b446e95e2a5d service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:44:35.764851 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0bcd5d44-5e15-49a5-930f-b446e95e2a5d service neutron] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:44:35.774166 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1418/2827] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 16:44:35 2022] GET /identity/v3/auth/tokens => generated 3327 bytes in 82 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 16:44:45.770005 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=26 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:44:50.584962 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1003a31-e281-40b0-ac25-c02bcd6eaf90 None None] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:44:50.585446 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1003a31-e281-40b0-ac25-c02bcd6eaf90 None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:44:50.585540 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1003a31-e281-40b0-ac25-c02bcd6eaf90 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:44:50.605682 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.auth.core [None req-d1003a31-e281-40b0-ac25-c02bcd6eaf90 None None] MFA Rules not processed for user `b6417e189b0d467480627cfd38331a9f`. Rule list: `[]` (Enabled: `True`). {{(pid=81438) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:44:50.606929 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-d1003a31-e281-40b0-ac25-c02bcd6eaf90 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:44:50.663749 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1410/2828] 10.222.0.50 () {58 vars in 958 bytes} [Tue Sep 13 16:44:50 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 80 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:45:00.656601 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=25 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:45:37.979157 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bcd5d44-5e15-49a5-930f-b446e95e2a5d service neutron] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:45:37.988483 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bcd5d44-5e15-49a5-930f-b446e95e2a5d service neutron] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:45:37.992053 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5109299e-884a-4fe1-b8a8-266e0dd390bb service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f436f07939d4855a21b195bc7566244', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:45:37.992648 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5109299e-884a-4fe1-b8a8-266e0dd390bb service neutron] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:45:37.992929 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5109299e-884a-4fe1-b8a8-266e0dd390bb service neutron] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:45:37.993201 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5109299e-884a-4fe1-b8a8-266e0dd390bb service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:45:37.994959 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-5109299e-884a-4fe1-b8a8-266e0dd390bb service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:45:38.057749 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5109299e-884a-4fe1-b8a8-266e0dd390bb service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:45:38.059326 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5109299e-884a-4fe1-b8a8-266e0dd390bb service neutron] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:45:38.069937 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1419/2829] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 16:45:37 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 91 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 16:45:41.236543 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1003a31-e281-40b0-ac25-c02bcd6eaf90 None None] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:45:41.238203 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-d1003a31-e281-40b0-ac25-c02bcd6eaf90 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:45:41.252987 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1003a31-e281-40b0-ac25-c02bcd6eaf90 None None] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:45:41.257599 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8f62d8e-6bab-4db9-8606-3684f2dc1956 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:45:41.258233 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8f62d8e-6bab-4db9-8606-3684f2dc1956 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:45:41.258514 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8f62d8e-6bab-4db9-8606-3684f2dc1956 admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:45:41.258792 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8f62d8e-6bab-4db9-8606-3684f2dc1956 admin admin] PATH_INFO: `/v3/users/c0fd4bbe430b4a8888f3be4b23982c5a` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:45:41.260088 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8f62d8e-6bab-4db9-8606-3684f2dc1956 admin admin] RBAC: Authorizing `identity:delete_user(user_id=c0fd4bbe430b4a8888f3be4b23982c5a)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:45:41.261999 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8f62d8e-6bab-4db9-8606-3684f2dc1956 admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:45:41.309131 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-e8f62d8e-6bab-4db9-8606-3684f2dc1956 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'c0fd4bbe430b4a8888f3be4b23982c5a'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:45:41.316663 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-e8f62d8e-6bab-4db9-8606-3684f2dc1956 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'c0fd4bbe430b4a8888f3be4b23982c5a'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:45:41.328870 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-e8f62d8e-6bab-4db9-8606-3684f2dc1956 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'c0fd4bbe430b4a8888f3be4b23982c5a'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:45:41.329518 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-e8f62d8e-6bab-4db9-8606-3684f2dc1956 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'c0fd4bbe430b4a8888f3be4b23982c5a'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:45:41.337349 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-e8f62d8e-6bab-4db9-8606-3684f2dc1956 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'c0fd4bbe430b4a8888f3be4b23982c5a'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:45:41.343206 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1411/2830] 10.222.0.50 () {58 vars in 1224 bytes} [Tue Sep 13 16:45:41 2022] DELETE /identity/v3/users/c0fd4bbe430b4a8888f3be4b23982c5a => generated 0 bytes in 107 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 16:45:41.515874 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5109299e-884a-4fe1-b8a8-266e0dd390bb service neutron] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:45:41.529372 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5109299e-884a-4fe1-b8a8-266e0dd390bb service neutron] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:45:41.589021 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71d52201-0c86-443a-b713-e599d6999769 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:45:41.589646 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71d52201-0c86-443a-b713-e599d6999769 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:45:41.589922 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71d52201-0c86-443a-b713-e599d6999769 admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:45:41.590195 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71d52201-0c86-443a-b713-e599d6999769 admin admin] PATH_INFO: `/v3/projects/859cfa45a18845f49e525ab4ed54b529` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:45:41.591333 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71d52201-0c86-443a-b713-e599d6999769 admin admin] RBAC: Authorizing `identity:delete_project(project_id=859cfa45a18845f49e525ab4ed54b529)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:45:41.593227 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71d52201-0c86-443a-b713-e599d6999769 admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:45:41.638992 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-71d52201-0c86-443a-b713-e599d6999769 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '859cfa45a18845f49e525ab4ed54b529', 'request_id': 'req-71d52201-0c86-443a-b713-e599d6999769'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:45:41.646044 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-71d52201-0c86-443a-b713-e599d6999769 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '859cfa45a18845f49e525ab4ed54b529', 'request_id': 'req-71d52201-0c86-443a-b713-e599d6999769'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:45:41.652153 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-71d52201-0c86-443a-b713-e599d6999769 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '859cfa45a18845f49e525ab4ed54b529', 'request_id': 'req-71d52201-0c86-443a-b713-e599d6999769'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:45:41.662355 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-71d52201-0c86-443a-b713-e599d6999769 admin admin] The token cache is being invalidate because project 859cfa45a18845f49e525ab4ed54b529 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81437) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 13 16:45:41.662641 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-71d52201-0c86-443a-b713-e599d6999769 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:45:41.663290 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-71d52201-0c86-443a-b713-e599d6999769 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:45:41.664759 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1420/2831] 10.222.0.50 () {58 vars in 1233 bytes} [Tue Sep 13 16:45:41 2022] DELETE /identity/v3/projects/859cfa45a18845f49e525ab4ed54b529 => generated 0 bytes in 150 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 16:45:41.671149 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a784c11e-d679-4619-844f-2b984672a101 None None] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:45:41.671437 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a784c11e-d679-4619-844f-2b984672a101 None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:45:41.671712 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a784c11e-d679-4619-844f-2b984672a101 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:45:41.692405 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.auth.core [None req-a784c11e-d679-4619-844f-2b984672a101 None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81438) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:45:41.693609 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-a784c11e-d679-4619-844f-2b984672a101 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:45:41.751634 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1412/2832] 10.222.0.50 () {58 vars in 958 bytes} [Tue Sep 13 16:45:41 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 82 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 13 16:45:41.760204 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25527a55-8e96-4b18-a3aa-fedb5feb9dc3 None None] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:45:41.760425 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25527a55-8e96-4b18-a3aa-fedb5feb9dc3 None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:45:41.760699 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25527a55-8e96-4b18-a3aa-fedb5feb9dc3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:45:41.780191 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.auth.core [None req-25527a55-8e96-4b18-a3aa-fedb5feb9dc3 None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81437) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:45:41.781383 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-25527a55-8e96-4b18-a3aa-fedb5feb9dc3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:45:41.789633 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1421/2833] 10.222.0.50 () {58 vars in 958 bytes} [Tue Sep 13 16:45:41 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:45:41.793620 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a784c11e-d679-4619-844f-2b984672a101 None None] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:45:41.803749 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a784c11e-d679-4619-844f-2b984672a101 None None] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:45:41.808334 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3497faf-d141-4036-9fd3-932f1de5bd5a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:45:41.808926 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3497faf-d141-4036-9fd3-932f1de5bd5a admin admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:45:41.809207 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3497faf-d141-4036-9fd3-932f1de5bd5a admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:45:41.809480 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3497faf-d141-4036-9fd3-932f1de5bd5a admin admin] PATH_INFO: `/v3/domains` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:45:41.810126 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3497faf-d141-4036-9fd3-932f1de5bd5a admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81438) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 16:45:41.810430 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3497faf-d141-4036-9fd3-932f1de5bd5a admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:45:41.812043 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3497faf-d141-4036-9fd3-932f1de5bd5a admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:45:41.824013 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1413/2834] 10.222.0.50 () {58 vars in 1153 bytes} [Tue Sep 13 16:45:41 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 16:45:41.828170 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25527a55-8e96-4b18-a3aa-fedb5feb9dc3 None None] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:45:41.837749 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25527a55-8e96-4b18-a3aa-fedb5feb9dc3 None None] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:45:41.842433 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5aa2f581-4a38-4f56-8476-aaa5012685be admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:45:41.843061 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5aa2f581-4a38-4f56-8476-aaa5012685be admin admin] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:45:41.843345 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5aa2f581-4a38-4f56-8476-aaa5012685be admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:45:41.843620 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5aa2f581-4a38-4f56-8476-aaa5012685be admin admin] PATH_INFO: `/v3/projects` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:45:41.844286 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5aa2f581-4a38-4f56-8476-aaa5012685be admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:45:41.846343 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5aa2f581-4a38-4f56-8476-aaa5012685be admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:45:41.862341 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:45:41.875230 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1422/2835] 10.222.0.50 () {60 vars in 1153 bytes} [Tue Sep 13 16:45:41 2022] POST /identity/v3/projects => generated 367 bytes in 47 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 16:45:41.880180 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3497faf-d141-4036-9fd3-932f1de5bd5a admin admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:45:41.890105 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3497faf-d141-4036-9fd3-932f1de5bd5a admin admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:45:41.946420 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19fc9d18-dd31-4a8a-b2b2-e92e8ee5b8cf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:45:41.947061 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19fc9d18-dd31-4a8a-b2b2-e92e8ee5b8cf admin admin] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:45:41.947343 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19fc9d18-dd31-4a8a-b2b2-e92e8ee5b8cf admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:45:41.947618 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19fc9d18-dd31-4a8a-b2b2-e92e8ee5b8cf admin admin] PATH_INFO: `/v3/users` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:45:41.948253 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19fc9d18-dd31-4a8a-b2b2-e92e8ee5b8cf admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:45:41.950180 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19fc9d18-dd31-4a8a-b2b2-e92e8ee5b8cf admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:45:41.958280 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:45:41.973410 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1414/2836] 10.222.0.50 () {60 vars in 1144 bytes} [Tue Sep 13 16:45:41 2022] POST /identity/v3/users => generated 284 bytes in 94 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 16:45:41.977569 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5aa2f581-4a38-4f56-8476-aaa5012685be admin admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:45:41.989350 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5aa2f581-4a38-4f56-8476-aaa5012685be admin admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:45:41.994050 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b60c4eea-13e5-407a-9b2a-83e0243fbdaf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:45:41.994702 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b60c4eea-13e5-407a-9b2a-83e0243fbdaf admin admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:45:41.995026 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b60c4eea-13e5-407a-9b2a-83e0243fbdaf admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:45:41.995303 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b60c4eea-13e5-407a-9b2a-83e0243fbdaf admin admin] PATH_INFO: `/v3/roles` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:45:41.995983 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b60c4eea-13e5-407a-9b2a-83e0243fbdaf admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:45:41.997654 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b60c4eea-13e5-407a-9b2a-83e0243fbdaf admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:45:42.008539 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1423/2837] 10.222.0.50 () {58 vars in 1122 bytes} [Tue Sep 13 16:45:41 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 16:45:42.012698 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19fc9d18-dd31-4a8a-b2b2-e92e8ee5b8cf admin admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:45:42.023123 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19fc9d18-dd31-4a8a-b2b2-e92e8ee5b8cf admin admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:45:42.027830 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d605d268-0701-45f1-80fc-36ae97331162 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:45:42.028530 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d605d268-0701-45f1-80fc-36ae97331162 admin admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:45:42.028810 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d605d268-0701-45f1-80fc-36ae97331162 admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:45:42.029078 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d605d268-0701-45f1-80fc-36ae97331162 admin admin] PATH_INFO: `/v3/roles` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:45:42.029710 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d605d268-0701-45f1-80fc-36ae97331162 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:45:42.031341 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d605d268-0701-45f1-80fc-36ae97331162 admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:45:42.046384 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1415/2838] 10.222.0.50 () {58 vars in 1122 bytes} [Tue Sep 13 16:45:42 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 16:45:42.048159 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b60c4eea-13e5-407a-9b2a-83e0243fbdaf admin admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:45:42.059285 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b60c4eea-13e5-407a-9b2a-83e0243fbdaf admin admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:45:42.064095 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-794165fe-d74a-45cf-a1ec-b019fd773e8f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:45:42.064995 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-794165fe-d74a-45cf-a1ec-b019fd773e8f admin admin] REQUEST_METHOD: `PUT` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:45:42.065501 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-794165fe-d74a-45cf-a1ec-b019fd773e8f admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:45:42.065595 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-794165fe-d74a-45cf-a1ec-b019fd773e8f admin admin] PATH_INFO: `/v3/projects/60abbd22f7584d80a9e2af6b8ec3594b/users/fbf8383a1b0940a186e7c05d0b1d4538/roles/15db4aaca3264ae29bde38903ba43471` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:45:42.081377 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-794165fe-d74a-45cf-a1ec-b019fd773e8f admin admin] RBAC: Authorizing `identity:create_grant(project_id=60abbd22f7584d80a9e2af6b8ec3594b, user_id=fbf8383a1b0940a186e7c05d0b1d4538, role_id=15db4aaca3264ae29bde38903ba43471)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:45:42.084089 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-794165fe-d74a-45cf-a1ec-b019fd773e8f admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:45:42.099062 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1424/2839] 10.222.0.50 () {60 vars in 1483 bytes} [Tue Sep 13 16:45:42 2022] PUT /identity/v3/projects/60abbd22f7584d80a9e2af6b8ec3594b/users/fbf8383a1b0940a186e7c05d0b1d4538/roles/15db4aaca3264ae29bde38903ba43471 => generated 0 bytes in 52 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 16:45:42.109335 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b49a3bed-a57f-413c-aa6e-7054f6f3cb28 None None] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:45:42.109703 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b49a3bed-a57f-413c-aa6e-7054f6f3cb28 None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:45:42.110108 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b49a3bed-a57f-413c-aa6e-7054f6f3cb28 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:45:42.138627 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.auth.core [None req-b49a3bed-a57f-413c-aa6e-7054f6f3cb28 None None] MFA Rules not processed for user `fbf8383a1b0940a186e7c05d0b1d4538`. Rule list: `[]` (Enabled: `True`). {{(pid=81438) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:45:42.139874 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-b49a3bed-a57f-413c-aa6e-7054f6f3cb28 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:45:42.203813 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1416/2840] 10.222.0.50 () {58 vars in 958 bytes} [Tue Sep 13 16:45:42 2022] POST /identity/v3/auth/tokens => generated 3244 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:45:42.214771 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-794165fe-d74a-45cf-a1ec-b019fd773e8f admin admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:45:42.216500 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-794165fe-d74a-45cf-a1ec-b019fd773e8f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:45:42.283828 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-794165fe-d74a-45cf-a1ec-b019fd773e8f admin admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:45:42.287414 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-104743a3-a879-496a-bba3-c137aa5b833f service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7fd6b5afd67b48f9b0f0597ccd4cf4df', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:45:42.288014 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-104743a3-a879-496a-bba3-c137aa5b833f service cinder] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:45:42.288292 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-104743a3-a879-496a-bba3-c137aa5b833f service cinder] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:45:42.288595 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-104743a3-a879-496a-bba3-c137aa5b833f service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:45:42.296677 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-104743a3-a879-496a-bba3-c137aa5b833f service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:45:42.298130 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-104743a3-a879-496a-bba3-c137aa5b833f service cinder] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:45:42.307432 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1425/2841] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 16:45:42 2022] GET /identity/v3/auth/tokens => generated 3244 bytes in 93 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 16:45:47.743630 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b49a3bed-a57f-413c-aa6e-7054f6f3cb28 None None] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:45:47.745032 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-b49a3bed-a57f-413c-aa6e-7054f6f3cb28 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:45:47.823628 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b49a3bed-a57f-413c-aa6e-7054f6f3cb28 None None] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:45:47.828670 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-325f7d8d-6f40-46c9-a566-fd530855727a service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '31480b61418340bd88583f034f67412e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:45:47.829271 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-325f7d8d-6f40-46c9-a566-fd530855727a service placement] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:45:47.829554 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-325f7d8d-6f40-46c9-a566-fd530855727a service placement] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:45:47.829833 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-325f7d8d-6f40-46c9-a566-fd530855727a service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:45:47.832061 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-325f7d8d-6f40-46c9-a566-fd530855727a service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:45:47.843377 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-325f7d8d-6f40-46c9-a566-fd530855727a service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:45:47.844942 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-325f7d8d-6f40-46c9-a566-fd530855727a service placement] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:45:47.854785 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1417/2842] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 16:45:47 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 112 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 16:45:50.656809 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-104743a3-a879-496a-bba3-c137aa5b833f service cinder] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:45:50.718760 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-104743a3-a879-496a-bba3-c137aa5b833f service cinder] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:45:50.723980 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50c0a750-d434-40fa-ae1e-cc11bad331a9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:45:50.724598 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50c0a750-d434-40fa-ae1e-cc11bad331a9 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:45:50.724875 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50c0a750-d434-40fa-ae1e-cc11bad331a9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:45:50.725263 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50c0a750-d434-40fa-ae1e-cc11bad331a9 admin admin] PATH_INFO: `/v3/users/fbf8383a1b0940a186e7c05d0b1d4538` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:45:50.726368 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50c0a750-d434-40fa-ae1e-cc11bad331a9 admin admin] RBAC: Authorizing `identity:delete_user(user_id=fbf8383a1b0940a186e7c05d0b1d4538)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:45:50.728326 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50c0a750-d434-40fa-ae1e-cc11bad331a9 admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:45:50.774911 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-50c0a750-d434-40fa-ae1e-cc11bad331a9 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'fbf8383a1b0940a186e7c05d0b1d4538'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:45:50.781988 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-50c0a750-d434-40fa-ae1e-cc11bad331a9 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'fbf8383a1b0940a186e7c05d0b1d4538'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:45:50.783987 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=90,keystone:INSERT=2,keystone:DELETE=8 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:45:50.796740 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-50c0a750-d434-40fa-ae1e-cc11bad331a9 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'fbf8383a1b0940a186e7c05d0b1d4538'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:45:50.797404 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-50c0a750-d434-40fa-ae1e-cc11bad331a9 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'fbf8383a1b0940a186e7c05d0b1d4538'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:45:50.804388 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-50c0a750-d434-40fa-ae1e-cc11bad331a9 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'fbf8383a1b0940a186e7c05d0b1d4538'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:45:50.810092 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1426/2843] 10.222.0.50 () {58 vars in 1224 bytes} [Tue Sep 13 16:45:50 2022] DELETE /identity/v3/users/fbf8383a1b0940a186e7c05d0b1d4538 => generated 0 bytes in 154 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 16:45:50.814635 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c98258f-8e5b-475c-a976-b1cd809a4e7f None None] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:45:50.815147 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c98258f-8e5b-475c-a976-b1cd809a4e7f None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:45:50.815247 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c98258f-8e5b-475c-a976-b1cd809a4e7f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:45:50.834559 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.auth.core [None req-5c98258f-8e5b-475c-a976-b1cd809a4e7f None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81438) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:45:50.835771 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-5c98258f-8e5b-475c-a976-b1cd809a4e7f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:45:50.879490 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:INSERT=3,keystone:SELECT=97 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:45:50.895735 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1418/2844] 10.222.0.50 () {58 vars in 958 bytes} [Tue Sep 13 16:45:50 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 82 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:45:50.905412 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50c0a750-d434-40fa-ae1e-cc11bad331a9 admin admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:45:50.906743 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-50c0a750-d434-40fa-ae1e-cc11bad331a9 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:45:50.966006 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50c0a750-d434-40fa-ae1e-cc11bad331a9 admin admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:45:50.970148 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a4d6d57-2db5-4f15-af3d-4c0ff8df0009 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f436f07939d4855a21b195bc7566244', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:45:50.970814 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a4d6d57-2db5-4f15-af3d-4c0ff8df0009 service neutron] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:45:50.971208 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a4d6d57-2db5-4f15-af3d-4c0ff8df0009 service neutron] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:45:50.971973 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a4d6d57-2db5-4f15-af3d-4c0ff8df0009 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:45:50.982339 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a4d6d57-2db5-4f15-af3d-4c0ff8df0009 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:45:50.983819 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a4d6d57-2db5-4f15-af3d-4c0ff8df0009 service neutron] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:45:50.993674 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1427/2845] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 16:45:50 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 89 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 16:45:51.267215 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c98258f-8e5b-475c-a976-b1cd809a4e7f None None] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:45:51.269164 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-5c98258f-8e5b-475c-a976-b1cd809a4e7f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:45:51.281138 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c98258f-8e5b-475c-a976-b1cd809a4e7f None None] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:45:51.286693 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f88404f-4cec-4926-9b29-107588dd7529 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:45:51.287367 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f88404f-4cec-4926-9b29-107588dd7529 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:45:51.287659 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f88404f-4cec-4926-9b29-107588dd7529 admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:45:51.287925 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f88404f-4cec-4926-9b29-107588dd7529 admin admin] PATH_INFO: `/v3/projects/60abbd22f7584d80a9e2af6b8ec3594b` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:45:51.289643 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f88404f-4cec-4926-9b29-107588dd7529 admin admin] RBAC: Authorizing `identity:delete_project(project_id=60abbd22f7584d80a9e2af6b8ec3594b)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:45:51.291621 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f88404f-4cec-4926-9b29-107588dd7529 admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:45:51.340928 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-3f88404f-4cec-4926-9b29-107588dd7529 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '60abbd22f7584d80a9e2af6b8ec3594b', 'request_id': 'req-3f88404f-4cec-4926-9b29-107588dd7529'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:45:51.348723 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-3f88404f-4cec-4926-9b29-107588dd7529 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '60abbd22f7584d80a9e2af6b8ec3594b', 'request_id': 'req-3f88404f-4cec-4926-9b29-107588dd7529'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:45:51.354276 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-3f88404f-4cec-4926-9b29-107588dd7529 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '60abbd22f7584d80a9e2af6b8ec3594b', 'request_id': 'req-3f88404f-4cec-4926-9b29-107588dd7529'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:45:51.365699 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-3f88404f-4cec-4926-9b29-107588dd7529 admin admin] The token cache is being invalidate because project 60abbd22f7584d80a9e2af6b8ec3594b was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81438) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 13 16:45:51.365989 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-3f88404f-4cec-4926-9b29-107588dd7529 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:45:51.366623 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-3f88404f-4cec-4926-9b29-107588dd7529 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:45:51.368112 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1419/2846] 10.222.0.50 () {58 vars in 1233 bytes} [Tue Sep 13 16:45:51 2022] DELETE /identity/v3/projects/60abbd22f7584d80a9e2af6b8ec3594b => generated 0 bytes in 102 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 16:45:51.372957 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3142bee6-d8eb-4de0-95c7-14fbc8f35073 None None] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:45:51.373293 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3142bee6-d8eb-4de0-95c7-14fbc8f35073 None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:45:51.373709 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3142bee6-d8eb-4de0-95c7-14fbc8f35073 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:45:51.393876 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.auth.core [None req-3142bee6-d8eb-4de0-95c7-14fbc8f35073 None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81437) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:45:51.395178 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-3142bee6-d8eb-4de0-95c7-14fbc8f35073 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:45:51.453532 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1428/2847] 10.222.0.50 () {58 vars in 958 bytes} [Tue Sep 13 16:45:51 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 81 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:45:51.461334 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c719c17-0857-46b7-a70d-0fc6669b1fde None None] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:45:51.461622 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c719c17-0857-46b7-a70d-0fc6669b1fde None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:45:51.461898 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c719c17-0857-46b7-a70d-0fc6669b1fde None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:45:51.482446 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.auth.core [None req-4c719c17-0857-46b7-a70d-0fc6669b1fde None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81438) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:45:51.483670 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-4c719c17-0857-46b7-a70d-0fc6669b1fde None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:45:51.492171 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1420/2848] 10.222.0.50 () {58 vars in 958 bytes} [Tue Sep 13 16:45:51 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:45:51.496425 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3142bee6-d8eb-4de0-95c7-14fbc8f35073 None None] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:45:51.506669 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3142bee6-d8eb-4de0-95c7-14fbc8f35073 None None] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:45:51.511184 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90f2d84f-e36e-48c2-9b90-a0b93f27243c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:45:51.511775 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90f2d84f-e36e-48c2-9b90-a0b93f27243c admin admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:45:51.512051 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90f2d84f-e36e-48c2-9b90-a0b93f27243c admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:45:51.512318 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90f2d84f-e36e-48c2-9b90-a0b93f27243c admin admin] PATH_INFO: `/v3/domains` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:45:51.512949 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90f2d84f-e36e-48c2-9b90-a0b93f27243c admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81437) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 16:45:51.513250 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90f2d84f-e36e-48c2-9b90-a0b93f27243c admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:45:51.514841 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90f2d84f-e36e-48c2-9b90-a0b93f27243c admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:45:51.526197 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1429/2849] 10.222.0.50 () {58 vars in 1153 bytes} [Tue Sep 13 16:45:51 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 16:45:51.530349 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c719c17-0857-46b7-a70d-0fc6669b1fde None None] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:45:51.539941 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c719c17-0857-46b7-a70d-0fc6669b1fde None None] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:45:51.544833 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5e7b0dc-b254-45be-a7b6-219736a92dbf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:45:51.545538 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5e7b0dc-b254-45be-a7b6-219736a92dbf admin admin] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:45:51.545891 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5e7b0dc-b254-45be-a7b6-219736a92dbf admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:45:51.546000 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5e7b0dc-b254-45be-a7b6-219736a92dbf admin admin] PATH_INFO: `/v3/projects` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:45:51.546755 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5e7b0dc-b254-45be-a7b6-219736a92dbf admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:45:51.548681 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5e7b0dc-b254-45be-a7b6-219736a92dbf admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:45:51.576569 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1421/2850] 10.222.0.50 () {60 vars in 1153 bytes} [Tue Sep 13 16:45:51 2022] POST /identity/v3/projects => generated 375 bytes in 47 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 16:45:51.580787 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90f2d84f-e36e-48c2-9b90-a0b93f27243c admin admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:45:51.590790 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90f2d84f-e36e-48c2-9b90-a0b93f27243c admin admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:45:51.647204 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23712c06-21b6-43e5-8682-2135227c9859 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:45:51.647802 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23712c06-21b6-43e5-8682-2135227c9859 admin admin] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:45:51.648082 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23712c06-21b6-43e5-8682-2135227c9859 admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:45:51.648354 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23712c06-21b6-43e5-8682-2135227c9859 admin admin] PATH_INFO: `/v3/users` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:45:51.648989 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23712c06-21b6-43e5-8682-2135227c9859 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:45:51.650921 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23712c06-21b6-43e5-8682-2135227c9859 admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:45:51.661256 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=96,keystone:DELETE=2,keystone:INSERT=2 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:45:51.674668 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1430/2851] 10.222.0.50 () {60 vars in 1144 bytes} [Tue Sep 13 16:45:51 2022] POST /identity/v3/users => generated 288 bytes in 95 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 16:45:51.679817 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5e7b0dc-b254-45be-a7b6-219736a92dbf admin admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:45:51.690007 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5e7b0dc-b254-45be-a7b6-219736a92dbf admin admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:45:51.695229 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48831c91-3a58-4871-abb6-354824af3c98 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:45:51.695825 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48831c91-3a58-4871-abb6-354824af3c98 admin admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:45:51.696132 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48831c91-3a58-4871-abb6-354824af3c98 admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:45:51.696406 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48831c91-3a58-4871-abb6-354824af3c98 admin admin] PATH_INFO: `/v3/roles` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:45:51.697040 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48831c91-3a58-4871-abb6-354824af3c98 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:45:51.698613 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48831c91-3a58-4871-abb6-354824af3c98 admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:45:51.709742 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1422/2852] 10.222.0.50 () {58 vars in 1122 bytes} [Tue Sep 13 16:45:51 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 16:45:51.713489 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23712c06-21b6-43e5-8682-2135227c9859 admin admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:45:51.723878 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23712c06-21b6-43e5-8682-2135227c9859 admin admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:45:51.728504 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17ba474b-db1c-4179-9042-54e1d826a711 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:45:51.729099 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17ba474b-db1c-4179-9042-54e1d826a711 admin admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:45:51.729386 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17ba474b-db1c-4179-9042-54e1d826a711 admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:45:51.729678 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17ba474b-db1c-4179-9042-54e1d826a711 admin admin] PATH_INFO: `/v3/roles` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:45:51.730299 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17ba474b-db1c-4179-9042-54e1d826a711 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:45:51.731925 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17ba474b-db1c-4179-9042-54e1d826a711 admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:45:51.742694 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1431/2853] 10.222.0.50 () {58 vars in 1122 bytes} [Tue Sep 13 16:45:51 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 13 16:45:51.746736 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48831c91-3a58-4871-abb6-354824af3c98 admin admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:45:51.757113 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48831c91-3a58-4871-abb6-354824af3c98 admin admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:45:51.761691 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab7f67d4-6377-4c79-9986-38f7887d0ecf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:45:51.762331 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab7f67d4-6377-4c79-9986-38f7887d0ecf admin admin] REQUEST_METHOD: `PUT` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:45:51.762613 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab7f67d4-6377-4c79-9986-38f7887d0ecf admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:45:51.762887 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab7f67d4-6377-4c79-9986-38f7887d0ecf admin admin] PATH_INFO: `/v3/projects/bd3a7ff18c784b09be4eebe9358046d0/users/8b2c4c17dc1a46c285bd6771267e9f67/roles/15db4aaca3264ae29bde38903ba43471` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:45:51.777867 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab7f67d4-6377-4c79-9986-38f7887d0ecf admin admin] RBAC: Authorizing `identity:create_grant(project_id=bd3a7ff18c784b09be4eebe9358046d0, user_id=8b2c4c17dc1a46c285bd6771267e9f67, role_id=15db4aaca3264ae29bde38903ba43471)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:45:51.780655 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab7f67d4-6377-4c79-9986-38f7887d0ecf admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:45:51.794219 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1423/2854] 10.222.0.50 () {60 vars in 1483 bytes} [Tue Sep 13 16:45:51 2022] PUT /identity/v3/projects/bd3a7ff18c784b09be4eebe9358046d0/users/8b2c4c17dc1a46c285bd6771267e9f67/roles/15db4aaca3264ae29bde38903ba43471 => generated 0 bytes in 48 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 13 16:45:51.804360 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d0bec1f-e0ed-4497-855d-0fc96dac98fb None None] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:45:51.804641 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d0bec1f-e0ed-4497-855d-0fc96dac98fb None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:45:51.804913 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d0bec1f-e0ed-4497-855d-0fc96dac98fb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:45:51.833268 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.auth.core [None req-4d0bec1f-e0ed-4497-855d-0fc96dac98fb None None] MFA Rules not processed for user `8b2c4c17dc1a46c285bd6771267e9f67`. Rule list: `[]` (Enabled: `True`). {{(pid=81437) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:45:51.834472 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-4d0bec1f-e0ed-4497-855d-0fc96dac98fb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:45:51.897067 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1432/2855] 10.222.0.50 () {58 vars in 958 bytes} [Tue Sep 13 16:45:51 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:45:51.912124 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab7f67d4-6377-4c79-9986-38f7887d0ecf admin admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:45:51.913859 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-ab7f67d4-6377-4c79-9986-38f7887d0ecf admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:45:51.987387 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab7f67d4-6377-4c79-9986-38f7887d0ecf admin admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:45:51.993454 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d26ace1c-92bf-41c6-bdb7-96edc0464273 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4e039b46c5e3445bbb39904b00c5430e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'admin', 'ResellerAdmin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:45:51.994063 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d26ace1c-92bf-41c6-bdb7-96edc0464273 service nova] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:45:51.994343 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d26ace1c-92bf-41c6-bdb7-96edc0464273 service nova] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:45:51.994616 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d26ace1c-92bf-41c6-bdb7-96edc0464273 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:45:51.999208 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:45:52.003884 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d26ace1c-92bf-41c6-bdb7-96edc0464273 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:45:52.005817 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d26ace1c-92bf-41c6-bdb7-96edc0464273 service nova] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:45:52.015830 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1424/2856] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 16:45:51 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 104 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 16:45:52.232422 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d0bec1f-e0ed-4497-855d-0fc96dac98fb None None] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:45:52.300047 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d0bec1f-e0ed-4497-855d-0fc96dac98fb None None] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:45:52.304760 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f60fb4b0-bc5b-47bb-875d-d727b27476bc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:45:52.305396 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f60fb4b0-bc5b-47bb-875d-d727b27476bc admin admin] REQUEST_METHOD: `DELETE` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:45:52.305676 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f60fb4b0-bc5b-47bb-875d-d727b27476bc admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:45:52.305952 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f60fb4b0-bc5b-47bb-875d-d727b27476bc admin admin] PATH_INFO: `/v3/users/8b2c4c17dc1a46c285bd6771267e9f67` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:45:52.307147 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f60fb4b0-bc5b-47bb-875d-d727b27476bc admin admin] RBAC: Authorizing `identity:delete_user(user_id=8b2c4c17dc1a46c285bd6771267e9f67)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:45:52.309086 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f60fb4b0-bc5b-47bb-875d-d727b27476bc admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:45:52.358963 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-f60fb4b0-bc5b-47bb-875d-d727b27476bc admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '8b2c4c17dc1a46c285bd6771267e9f67'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:45:52.368360 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-f60fb4b0-bc5b-47bb-875d-d727b27476bc admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '8b2c4c17dc1a46c285bd6771267e9f67'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:45:52.381712 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-f60fb4b0-bc5b-47bb-875d-d727b27476bc admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '8b2c4c17dc1a46c285bd6771267e9f67'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:45:52.382418 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-f60fb4b0-bc5b-47bb-875d-d727b27476bc admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '8b2c4c17dc1a46c285bd6771267e9f67'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:45:52.385500 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:INSERT=2,keystone:SELECT=88,keystone:DELETE=10 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:45:52.392574 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-f60fb4b0-bc5b-47bb-875d-d727b27476bc admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '8b2c4c17dc1a46c285bd6771267e9f67'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:45:52.400589 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1433/2857] 10.222.0.50 () {58 vars in 1224 bytes} [Tue Sep 13 16:45:52 2022] DELETE /identity/v3/users/8b2c4c17dc1a46c285bd6771267e9f67 => generated 0 bytes in 169 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 16:45:52.405309 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ede3c0b4-e40c-48bf-993b-5662cfa379c8 None None] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:45:52.405594 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ede3c0b4-e40c-48bf-993b-5662cfa379c8 None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:45:52.405867 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ede3c0b4-e40c-48bf-993b-5662cfa379c8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:45:52.425572 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.auth.core [None req-ede3c0b4-e40c-48bf-993b-5662cfa379c8 None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81438) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:45:52.426796 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-ede3c0b4-e40c-48bf-993b-5662cfa379c8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:45:52.491032 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1425/2858] 10.222.0.50 () {58 vars in 958 bytes} [Tue Sep 13 16:45:52 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:45:52.497715 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f60fb4b0-bc5b-47bb-875d-d727b27476bc admin admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:45:52.499289 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-f60fb4b0-bc5b-47bb-875d-d727b27476bc admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:45:52.562677 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f60fb4b0-bc5b-47bb-875d-d727b27476bc admin admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:45:52.567184 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-217f337e-bb02-4418-ae16-8fbbf9a2e63a service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f436f07939d4855a21b195bc7566244', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:45:52.567798 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-217f337e-bb02-4418-ae16-8fbbf9a2e63a service neutron] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:45:52.568079 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-217f337e-bb02-4418-ae16-8fbbf9a2e63a service neutron] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:45:52.568357 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-217f337e-bb02-4418-ae16-8fbbf9a2e63a service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:45:52.577545 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-217f337e-bb02-4418-ae16-8fbbf9a2e63a service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:45:52.579019 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-217f337e-bb02-4418-ae16-8fbbf9a2e63a service neutron] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:45:52.590016 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1434/2859] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 16:45:52 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 93 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 16:45:53.317813 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ede3c0b4-e40c-48bf-993b-5662cfa379c8 None None] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:45:53.319328 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-ede3c0b4-e40c-48bf-993b-5662cfa379c8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:45:53.331835 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ede3c0b4-e40c-48bf-993b-5662cfa379c8 None None] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:45:53.336619 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d954b3ec-cd5a-4bf2-a8b9-7bfa911b3173 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:45:53.337255 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d954b3ec-cd5a-4bf2-a8b9-7bfa911b3173 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:45:53.337540 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d954b3ec-cd5a-4bf2-a8b9-7bfa911b3173 admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:45:53.337816 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d954b3ec-cd5a-4bf2-a8b9-7bfa911b3173 admin admin] PATH_INFO: `/v3/projects/bd3a7ff18c784b09be4eebe9358046d0` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:45:53.338979 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d954b3ec-cd5a-4bf2-a8b9-7bfa911b3173 admin admin] RBAC: Authorizing `identity:delete_project(project_id=bd3a7ff18c784b09be4eebe9358046d0)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:45:53.340895 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d954b3ec-cd5a-4bf2-a8b9-7bfa911b3173 admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:45:53.392474 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-d954b3ec-cd5a-4bf2-a8b9-7bfa911b3173 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'bd3a7ff18c784b09be4eebe9358046d0', 'request_id': 'req-d954b3ec-cd5a-4bf2-a8b9-7bfa911b3173'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:45:53.399877 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-d954b3ec-cd5a-4bf2-a8b9-7bfa911b3173 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'bd3a7ff18c784b09be4eebe9358046d0', 'request_id': 'req-d954b3ec-cd5a-4bf2-a8b9-7bfa911b3173'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:45:53.406126 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-d954b3ec-cd5a-4bf2-a8b9-7bfa911b3173 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'bd3a7ff18c784b09be4eebe9358046d0', 'request_id': 'req-d954b3ec-cd5a-4bf2-a8b9-7bfa911b3173'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:45:53.418929 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-d954b3ec-cd5a-4bf2-a8b9-7bfa911b3173 admin admin] The token cache is being invalidate because project bd3a7ff18c784b09be4eebe9358046d0 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81438) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 13 16:45:53.419280 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-d954b3ec-cd5a-4bf2-a8b9-7bfa911b3173 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:45:53.419998 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-d954b3ec-cd5a-4bf2-a8b9-7bfa911b3173 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:45:53.421499 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1426/2860] 10.222.0.50 () {58 vars in 1233 bytes} [Tue Sep 13 16:45:53 2022] DELETE /identity/v3/projects/bd3a7ff18c784b09be4eebe9358046d0 => generated 0 bytes in 104 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 16:45:53.426813 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a680b22-f713-4f58-a742-38b80a3f9b59 None None] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:45:53.427333 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a680b22-f713-4f58-a742-38b80a3f9b59 None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:45:53.427450 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a680b22-f713-4f58-a742-38b80a3f9b59 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:45:53.449183 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.auth.core [None req-1a680b22-f713-4f58-a742-38b80a3f9b59 None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81437) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:45:53.450310 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-1a680b22-f713-4f58-a742-38b80a3f9b59 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:45:53.511134 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1435/2861] 10.222.0.50 () {58 vars in 958 bytes} [Tue Sep 13 16:45:53 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:45:53.520537 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30d00d41-bf82-4864-b484-f15e51832820 None None] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:45:53.520819 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30d00d41-bf82-4864-b484-f15e51832820 None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:45:53.521096 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30d00d41-bf82-4864-b484-f15e51832820 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:45:53.541171 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.auth.core [None req-30d00d41-bf82-4864-b484-f15e51832820 None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81438) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:45:53.542372 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-30d00d41-bf82-4864-b484-f15e51832820 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:45:53.551190 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1427/2862] 10.222.0.50 () {58 vars in 958 bytes} [Tue Sep 13 16:45:53 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:45:53.555559 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a680b22-f713-4f58-a742-38b80a3f9b59 None None] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:45:53.565376 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a680b22-f713-4f58-a742-38b80a3f9b59 None None] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:45:53.570836 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a6182bf-f260-4487-a50d-972c9e11d5b9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:45:53.571477 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a6182bf-f260-4487-a50d-972c9e11d5b9 admin admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:45:53.571763 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a6182bf-f260-4487-a50d-972c9e11d5b9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:45:53.572034 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a6182bf-f260-4487-a50d-972c9e11d5b9 admin admin] PATH_INFO: `/v3/domains` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:45:53.572684 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a6182bf-f260-4487-a50d-972c9e11d5b9 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81437) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 16:45:53.572993 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a6182bf-f260-4487-a50d-972c9e11d5b9 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:45:53.574750 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a6182bf-f260-4487-a50d-972c9e11d5b9 admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:45:53.586998 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1436/2863] 10.222.0.50 () {58 vars in 1153 bytes} [Tue Sep 13 16:45:53 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 16:45:53.591454 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30d00d41-bf82-4864-b484-f15e51832820 None None] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:45:53.600651 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30d00d41-bf82-4864-b484-f15e51832820 None None] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:45:53.605309 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7107bdbc-4690-4eba-85f1-c95c6c38cc47 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:45:53.605905 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7107bdbc-4690-4eba-85f1-c95c6c38cc47 admin admin] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:45:53.606187 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7107bdbc-4690-4eba-85f1-c95c6c38cc47 admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:45:53.606460 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7107bdbc-4690-4eba-85f1-c95c6c38cc47 admin admin] PATH_INFO: `/v3/projects` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:45:53.607141 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7107bdbc-4690-4eba-85f1-c95c6c38cc47 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:45:53.609045 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7107bdbc-4690-4eba-85f1-c95c6c38cc47 admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:45:53.641873 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1428/2864] 10.222.0.50 () {60 vars in 1153 bytes} [Tue Sep 13 16:45:53 2022] POST /identity/v3/projects => generated 357 bytes in 48 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 16:45:53.643113 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a6182bf-f260-4487-a50d-972c9e11d5b9 admin admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:45:53.653258 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a6182bf-f260-4487-a50d-972c9e11d5b9 admin admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:45:53.713094 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cb9b232-5c18-4ee4-af6d-734ed3ac7889 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:45:53.713702 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cb9b232-5c18-4ee4-af6d-734ed3ac7889 admin admin] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:45:53.713981 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cb9b232-5c18-4ee4-af6d-734ed3ac7889 admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:45:53.714255 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cb9b232-5c18-4ee4-af6d-734ed3ac7889 admin admin] PATH_INFO: `/v3/users` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:45:53.715048 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0cb9b232-5c18-4ee4-af6d-734ed3ac7889 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:45:53.717170 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0cb9b232-5c18-4ee4-af6d-734ed3ac7889 admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:45:53.743788 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1437/2865] 10.222.0.50 () {60 vars in 1144 bytes} [Tue Sep 13 16:45:53 2022] POST /identity/v3/users => generated 279 bytes in 98 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 16:45:53.745321 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7107bdbc-4690-4eba-85f1-c95c6c38cc47 admin admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:45:53.756546 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7107bdbc-4690-4eba-85f1-c95c6c38cc47 admin admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:45:53.761241 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9114cf3-4e15-45d6-9e97-1de71b384179 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:45:53.761903 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9114cf3-4e15-45d6-9e97-1de71b384179 admin admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:45:53.762255 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9114cf3-4e15-45d6-9e97-1de71b384179 admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:45:53.762642 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9114cf3-4e15-45d6-9e97-1de71b384179 admin admin] PATH_INFO: `/v3/roles` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:45:53.763386 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9114cf3-4e15-45d6-9e97-1de71b384179 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:45:53.765028 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9114cf3-4e15-45d6-9e97-1de71b384179 admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:45:53.775584 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1429/2866] 10.222.0.50 () {58 vars in 1122 bytes} [Tue Sep 13 16:45:53 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 16:45:53.780484 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cb9b232-5c18-4ee4-af6d-734ed3ac7889 admin admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:45:53.791503 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cb9b232-5c18-4ee4-af6d-734ed3ac7889 admin admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:45:53.796188 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f20350b3-7c13-4bda-b702-4de1d46cc281 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:45:53.796809 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f20350b3-7c13-4bda-b702-4de1d46cc281 admin admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:45:53.797076 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f20350b3-7c13-4bda-b702-4de1d46cc281 admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:45:53.797349 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f20350b3-7c13-4bda-b702-4de1d46cc281 admin admin] PATH_INFO: `/v3/roles` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:45:53.798109 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f20350b3-7c13-4bda-b702-4de1d46cc281 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:45:53.800308 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f20350b3-7c13-4bda-b702-4de1d46cc281 admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:45:53.806341 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:45:53.813499 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1438/2867] 10.222.0.50 () {58 vars in 1122 bytes} [Tue Sep 13 16:45:53 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 16:45:53.817993 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9114cf3-4e15-45d6-9e97-1de71b384179 admin admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:45:53.828471 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9114cf3-4e15-45d6-9e97-1de71b384179 admin admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:45:53.833577 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f268a055-97b5-4a0a-9969-107783b29386 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:45:53.834237 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f268a055-97b5-4a0a-9969-107783b29386 admin admin] REQUEST_METHOD: `PUT` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:45:53.834521 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f268a055-97b5-4a0a-9969-107783b29386 admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:45:53.834798 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f268a055-97b5-4a0a-9969-107783b29386 admin admin] PATH_INFO: `/v3/projects/a1fcf85aa12a42c99c427cc2dc9476e2/users/1e1b4b76596c4c7aa9c1a7e26c055772/roles/15db4aaca3264ae29bde38903ba43471` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:45:53.850585 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f268a055-97b5-4a0a-9969-107783b29386 admin admin] RBAC: Authorizing `identity:create_grant(project_id=a1fcf85aa12a42c99c427cc2dc9476e2, user_id=1e1b4b76596c4c7aa9c1a7e26c055772, role_id=15db4aaca3264ae29bde38903ba43471)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:45:53.853273 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f268a055-97b5-4a0a-9969-107783b29386 admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:45:53.868050 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1430/2868] 10.222.0.50 () {60 vars in 1483 bytes} [Tue Sep 13 16:45:53 2022] PUT /identity/v3/projects/a1fcf85aa12a42c99c427cc2dc9476e2/users/1e1b4b76596c4c7aa9c1a7e26c055772/roles/15db4aaca3264ae29bde38903ba43471 => generated 0 bytes in 51 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 16:45:53.878580 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c42c3a9f-e73b-4958-918e-86d1250aea16 None None] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:45:53.878868 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c42c3a9f-e73b-4958-918e-86d1250aea16 None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:45:53.879178 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c42c3a9f-e73b-4958-918e-86d1250aea16 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:45:53.908655 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.auth.core [None req-c42c3a9f-e73b-4958-918e-86d1250aea16 None None] MFA Rules not processed for user `1e1b4b76596c4c7aa9c1a7e26c055772`. Rule list: `[]` (Enabled: `True`). {{(pid=81437) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:45:53.911146 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-c42c3a9f-e73b-4958-918e-86d1250aea16 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:45:53.977792 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1439/2869] 10.222.0.50 () {58 vars in 958 bytes} [Tue Sep 13 16:45:53 2022] POST /identity/v3/auth/tokens => generated 3234 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:45:53.982425 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f268a055-97b5-4a0a-9969-107783b29386 admin admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:45:53.998359 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:45:54.049374 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f268a055-97b5-4a0a-9969-107783b29386 admin admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:45:54.055432 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-672e36a1-8b41-4f3d-97f2-d1df47927a78 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:45:54.056063 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-672e36a1-8b41-4f3d-97f2-d1df47927a78 admin admin] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:45:54.056333 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-672e36a1-8b41-4f3d-97f2-d1df47927a78 admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:45:54.056725 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-672e36a1-8b41-4f3d-97f2-d1df47927a78 admin admin] PATH_INFO: `/v3/projects` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:45:54.057371 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-672e36a1-8b41-4f3d-97f2-d1df47927a78 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:45:54.059337 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-672e36a1-8b41-4f3d-97f2-d1df47927a78 admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:45:54.087120 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1431/2870] 10.222.0.50 () {60 vars in 1153 bytes} [Tue Sep 13 16:45:53 2022] POST /identity/v3/projects => generated 355 bytes in 105 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 16:45:54.091251 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c42c3a9f-e73b-4958-918e-86d1250aea16 None None] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:45:54.102701 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c42c3a9f-e73b-4958-918e-86d1250aea16 None None] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:45:54.179416 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9ee9c5c-a586-41ac-a087-873ad66b5399 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:45:54.179870 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9ee9c5c-a586-41ac-a087-873ad66b5399 admin admin] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:45:54.180152 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9ee9c5c-a586-41ac-a087-873ad66b5399 admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:45:54.180427 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9ee9c5c-a586-41ac-a087-873ad66b5399 admin admin] PATH_INFO: `/v3/users` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:45:54.181065 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9ee9c5c-a586-41ac-a087-873ad66b5399 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:45:54.184614 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9ee9c5c-a586-41ac-a087-873ad66b5399 admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:45:54.206649 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1440/2871] 10.222.0.50 () {60 vars in 1144 bytes} [Tue Sep 13 16:45:54 2022] POST /identity/v3/users => generated 284 bytes in 116 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 16:45:54.210912 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-672e36a1-8b41-4f3d-97f2-d1df47927a78 admin admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:45:54.224005 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-672e36a1-8b41-4f3d-97f2-d1df47927a78 admin admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:45:54.228849 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-baf349e0-3dd0-46a3-9369-456e42eed8d6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:45:54.229533 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-baf349e0-3dd0-46a3-9369-456e42eed8d6 admin admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:45:54.229802 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-baf349e0-3dd0-46a3-9369-456e42eed8d6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:45:54.230081 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-baf349e0-3dd0-46a3-9369-456e42eed8d6 admin admin] PATH_INFO: `/v3/roles` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:45:54.230725 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-baf349e0-3dd0-46a3-9369-456e42eed8d6 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:45:54.232497 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-baf349e0-3dd0-46a3-9369-456e42eed8d6 admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:45:54.245761 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1432/2872] 10.222.0.50 () {58 vars in 1122 bytes} [Tue Sep 13 16:45:54 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 16:45:54.249941 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9ee9c5c-a586-41ac-a087-873ad66b5399 admin admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:45:54.264235 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9ee9c5c-a586-41ac-a087-873ad66b5399 admin admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:45:54.270396 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a492d0b5-fb61-4733-a540-2e26958e763d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:45:54.270564 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a492d0b5-fb61-4733-a540-2e26958e763d admin admin] REQUEST_METHOD: `PUT` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:45:54.270763 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a492d0b5-fb61-4733-a540-2e26958e763d admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:45:54.271061 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a492d0b5-fb61-4733-a540-2e26958e763d admin admin] PATH_INFO: `/v3/projects/c9acaa93110c4f119215036a7107f59a/users/b2f304bdba5c4bb8922be96ce5695bde/roles/031d4e1a23e34f71858fc4a6dfbd4ccb` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:45:54.288893 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a492d0b5-fb61-4733-a540-2e26958e763d admin admin] RBAC: Authorizing `identity:create_grant(project_id=c9acaa93110c4f119215036a7107f59a, user_id=b2f304bdba5c4bb8922be96ce5695bde, role_id=031d4e1a23e34f71858fc4a6dfbd4ccb)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:45:54.291685 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a492d0b5-fb61-4733-a540-2e26958e763d admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:45:54.308806 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1441/2873] 10.222.0.50 () {60 vars in 1483 bytes} [Tue Sep 13 16:45:54 2022] PUT /identity/v3/projects/c9acaa93110c4f119215036a7107f59a/users/b2f304bdba5c4bb8922be96ce5695bde/roles/031d4e1a23e34f71858fc4a6dfbd4ccb => generated 0 bytes in 59 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 16:45:54.313150 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-baf349e0-3dd0-46a3-9369-456e42eed8d6 admin admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:45:54.324543 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-baf349e0-3dd0-46a3-9369-456e42eed8d6 admin admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:45:54.396909 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d49b1b15-920e-4857-8a0f-f19d1990eb7c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:45:54.397528 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d49b1b15-920e-4857-8a0f-f19d1990eb7c admin admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:45:54.397814 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d49b1b15-920e-4857-8a0f-f19d1990eb7c admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:45:54.398134 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d49b1b15-920e-4857-8a0f-f19d1990eb7c admin admin] PATH_INFO: `/v3/roles` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:45:54.398750 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d49b1b15-920e-4857-8a0f-f19d1990eb7c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:45:54.400359 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d49b1b15-920e-4857-8a0f-f19d1990eb7c admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:45:54.412841 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1433/2874] 10.222.0.50 () {58 vars in 1122 bytes} [Tue Sep 13 16:45:54 2022] GET /identity/v3/roles => generated 1420 bytes in 100 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 16:45:54.416764 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a492d0b5-fb61-4733-a540-2e26958e763d admin admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:45:54.429007 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a492d0b5-fb61-4733-a540-2e26958e763d admin admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:45:54.433865 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c694324-9597-4665-8df2-423afff8a0e8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:45:54.434615 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c694324-9597-4665-8df2-423afff8a0e8 admin admin] REQUEST_METHOD: `PUT` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:45:54.434994 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c694324-9597-4665-8df2-423afff8a0e8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:45:54.435342 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c694324-9597-4665-8df2-423afff8a0e8 admin admin] PATH_INFO: `/v3/projects/c9acaa93110c4f119215036a7107f59a/users/b2f304bdba5c4bb8922be96ce5695bde/roles/031d4e1a23e34f71858fc4a6dfbd4ccb` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:45:54.438019 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c694324-9597-4665-8df2-423afff8a0e8 admin admin] RBAC: Authorizing `identity:create_grant(project_id=c9acaa93110c4f119215036a7107f59a, user_id=b2f304bdba5c4bb8922be96ce5695bde, role_id=031d4e1a23e34f71858fc4a6dfbd4ccb)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:45:54.441591 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c694324-9597-4665-8df2-423afff8a0e8 admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:45:54.453371 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1442/2875] 10.222.0.50 () {60 vars in 1483 bytes} [Tue Sep 13 16:45:54 2022] PUT /identity/v3/projects/c9acaa93110c4f119215036a7107f59a/users/b2f304bdba5c4bb8922be96ce5695bde/roles/031d4e1a23e34f71858fc4a6dfbd4ccb => generated 0 bytes in 37 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 16:45:54.463974 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-241377c8-79e1-447c-b3bd-1266b78a3058 None None] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:45:54.464293 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-241377c8-79e1-447c-b3bd-1266b78a3058 None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:45:54.464744 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-241377c8-79e1-447c-b3bd-1266b78a3058 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:45:54.496017 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.auth.core [None req-241377c8-79e1-447c-b3bd-1266b78a3058 None None] MFA Rules not processed for user `b2f304bdba5c4bb8922be96ce5695bde`. Rule list: `[]` (Enabled: `True`). {{(pid=81438) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:45:54.497242 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-241377c8-79e1-447c-b3bd-1266b78a3058 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:45:54.575370 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1434/2876] 10.222.0.50 () {58 vars in 958 bytes} [Tue Sep 13 16:45:54 2022] POST /identity/v3/auth/tokens => generated 3299 bytes in 112 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:45:54.587630 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c694324-9597-4665-8df2-423afff8a0e8 admin admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:45:54.589493 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-2c694324-9597-4665-8df2-423afff8a0e8 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:45:54.638595 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:45:54.651626 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c694324-9597-4665-8df2-423afff8a0e8 admin admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:45:54.655335 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61f068c5-1f09-4738-9739-1c62d9016774 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7fd6b5afd67b48f9b0f0597ccd4cf4df', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:45:54.655945 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61f068c5-1f09-4738-9739-1c62d9016774 service cinder] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:45:54.656224 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61f068c5-1f09-4738-9739-1c62d9016774 service cinder] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:45:54.656500 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61f068c5-1f09-4738-9739-1c62d9016774 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:45:54.666644 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61f068c5-1f09-4738-9739-1c62d9016774 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:45:54.668128 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61f068c5-1f09-4738-9739-1c62d9016774 service cinder] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:45:54.678389 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1443/2877] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 16:45:54 2022] GET /identity/v3/auth/tokens => generated 3299 bytes in 91 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 16:45:54.759992 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-241377c8-79e1-447c-b3bd-1266b78a3058 None None] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:45:54.770945 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-241377c8-79e1-447c-b3bd-1266b78a3058 None None] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:45:54.774673 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85e3f339-1a17-4d86-9d16-a5996717fc12 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7fd6b5afd67b48f9b0f0597ccd4cf4df', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:45:54.775326 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85e3f339-1a17-4d86-9d16-a5996717fc12 service cinder] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:45:54.775616 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85e3f339-1a17-4d86-9d16-a5996717fc12 service cinder] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:45:54.775884 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85e3f339-1a17-4d86-9d16-a5996717fc12 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:45:54.782117 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:45:54.855311 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85e3f339-1a17-4d86-9d16-a5996717fc12 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:45:54.856776 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85e3f339-1a17-4d86-9d16-a5996717fc12 service cinder] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:45:54.866315 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1435/2878] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 16:45:54 2022] GET /identity/v3/auth/tokens => generated 3234 bytes in 107 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 16:46:00.391468 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38208576-a20a-4ded-ad81-39575aaa29fa None None] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:46:00.391994 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38208576-a20a-4ded-ad81-39575aaa29fa None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:46:00.392078 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38208576-a20a-4ded-ad81-39575aaa29fa None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:46:00.413222 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.auth.core [None req-38208576-a20a-4ded-ad81-39575aaa29fa None None] MFA Rules not processed for user `f8364b1a190d4218bf353a89bdb81448`. Rule list: `[]` (Enabled: `True`). {{(pid=81437) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:46:00.414507 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-38208576-a20a-4ded-ad81-39575aaa29fa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:46:00.470809 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1444/2879] 10.222.0.50 () {58 vars in 967 bytes} [Tue Sep 13 16:46:00 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 80 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:46:00.511462 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85e3f339-1a17-4d86-9d16-a5996717fc12 service cinder] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:46:00.512868 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-85e3f339-1a17-4d86-9d16-a5996717fc12 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:46:00.583716 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85e3f339-1a17-4d86-9d16-a5996717fc12 service cinder] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:46:00.588728 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0810e59-6bdd-4bd8-9b0a-280eead09da5 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b00ab0d6321d483188bc69deaddcdbbc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:46:00.589335 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0810e59-6bdd-4bd8-9b0a-280eead09da5 service swift] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:46:00.589614 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0810e59-6bdd-4bd8-9b0a-280eead09da5 service swift] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:46:00.589886 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0810e59-6bdd-4bd8-9b0a-280eead09da5 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:46:00.598407 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0810e59-6bdd-4bd8-9b0a-280eead09da5 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:46:00.600008 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0810e59-6bdd-4bd8-9b0a-280eead09da5 service swift] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:46:00.608003 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1436/2880] 10.222.0.50 () {60 vars in 1384 bytes} [Tue Sep 13 16:46:00 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 97 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Sep 13 16:46:07.100218 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38208576-a20a-4ded-ad81-39575aaa29fa None None] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:46:07.101779 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-38208576-a20a-4ded-ad81-39575aaa29fa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:46:07.175189 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38208576-a20a-4ded-ad81-39575aaa29fa None None] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:46:07.180304 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50797607-c453-4a35-9e7c-6b9f129fefff admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:46:07.180940 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50797607-c453-4a35-9e7c-6b9f129fefff admin admin] REQUEST_METHOD: `DELETE` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:46:07.181238 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50797607-c453-4a35-9e7c-6b9f129fefff admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:46:07.181645 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50797607-c453-4a35-9e7c-6b9f129fefff admin admin] PATH_INFO: `/v3/users/91ce9cad1eab41efb4da9503c23ad073` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:46:07.182826 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50797607-c453-4a35-9e7c-6b9f129fefff admin admin] RBAC: Authorizing `identity:delete_user(user_id=91ce9cad1eab41efb4da9503c23ad073)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:46:07.184916 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50797607-c453-4a35-9e7c-6b9f129fefff admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:46:07.238316 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-50797607-c453-4a35-9e7c-6b9f129fefff admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '91ce9cad1eab41efb4da9503c23ad073'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:46:07.244996 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-50797607-c453-4a35-9e7c-6b9f129fefff admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '91ce9cad1eab41efb4da9503c23ad073'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:46:07.257317 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-50797607-c453-4a35-9e7c-6b9f129fefff admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '91ce9cad1eab41efb4da9503c23ad073'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:46:07.257967 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-50797607-c453-4a35-9e7c-6b9f129fefff admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '91ce9cad1eab41efb4da9503c23ad073'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:46:07.264822 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-50797607-c453-4a35-9e7c-6b9f129fefff admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '91ce9cad1eab41efb4da9503c23ad073'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:46:07.271355 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1445/2881] 10.222.0.50 () {58 vars in 1224 bytes} [Tue Sep 13 16:46:07 2022] DELETE /identity/v3/users/91ce9cad1eab41efb4da9503c23ad073 => generated 0 bytes in 172 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 16:46:07.447238 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0810e59-6bdd-4bd8-9b0a-280eead09da5 service swift] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:46:07.460358 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0810e59-6bdd-4bd8-9b0a-280eead09da5 service swift] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:46:07.517671 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-564394a4-4f15-4705-9286-2008ece8daf3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:46:07.518300 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-564394a4-4f15-4705-9286-2008ece8daf3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:46:07.518576 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-564394a4-4f15-4705-9286-2008ece8daf3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:46:07.518847 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-564394a4-4f15-4705-9286-2008ece8daf3 admin admin] PATH_INFO: `/v3/projects/08f6037f42c34be7b2407c258d8885be` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:46:07.519970 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-564394a4-4f15-4705-9286-2008ece8daf3 admin admin] RBAC: Authorizing `identity:delete_project(project_id=08f6037f42c34be7b2407c258d8885be)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:46:07.522535 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-564394a4-4f15-4705-9286-2008ece8daf3 admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:46:07.561068 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=97,keystone:DELETE=3 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:46:07.569791 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-564394a4-4f15-4705-9286-2008ece8daf3 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '08f6037f42c34be7b2407c258d8885be', 'request_id': 'req-564394a4-4f15-4705-9286-2008ece8daf3'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:46:07.577262 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-564394a4-4f15-4705-9286-2008ece8daf3 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '08f6037f42c34be7b2407c258d8885be', 'request_id': 'req-564394a4-4f15-4705-9286-2008ece8daf3'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:46:07.583175 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-564394a4-4f15-4705-9286-2008ece8daf3 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '08f6037f42c34be7b2407c258d8885be', 'request_id': 'req-564394a4-4f15-4705-9286-2008ece8daf3'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:46:07.594507 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-564394a4-4f15-4705-9286-2008ece8daf3 admin admin] The token cache is being invalidate because project 08f6037f42c34be7b2407c258d8885be was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81438) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 13 16:46:07.594805 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-564394a4-4f15-4705-9286-2008ece8daf3 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:46:07.595466 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-564394a4-4f15-4705-9286-2008ece8daf3 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:46:07.596949 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1437/2882] 10.222.0.50 () {58 vars in 1233 bytes} [Tue Sep 13 16:46:07 2022] DELETE /identity/v3/projects/08f6037f42c34be7b2407c258d8885be => generated 0 bytes in 151 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 16:46:07.601164 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50797607-c453-4a35-9e7c-6b9f129fefff admin admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:46:07.602567 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-50797607-c453-4a35-9e7c-6b9f129fefff admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:46:07.615487 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50797607-c453-4a35-9e7c-6b9f129fefff admin admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:46:07.620059 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:46:07.673907 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5103f6d9-6c38-4ce1-acef-1fce54a05193 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:46:07.674543 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5103f6d9-6c38-4ce1-acef-1fce54a05193 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:46:07.674827 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5103f6d9-6c38-4ce1-acef-1fce54a05193 admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:46:07.675143 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5103f6d9-6c38-4ce1-acef-1fce54a05193 admin admin] PATH_INFO: `/v3/users/2d30f26e52424a7e92a0099fbc480683` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:46:07.676283 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5103f6d9-6c38-4ce1-acef-1fce54a05193 admin admin] RBAC: Authorizing `identity:delete_user(user_id=2d30f26e52424a7e92a0099fbc480683)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:46:07.678825 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5103f6d9-6c38-4ce1-acef-1fce54a05193 admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:46:07.724274 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-5103f6d9-6c38-4ce1-acef-1fce54a05193 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '2d30f26e52424a7e92a0099fbc480683'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:46:07.731576 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-5103f6d9-6c38-4ce1-acef-1fce54a05193 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '2d30f26e52424a7e92a0099fbc480683'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:46:07.744702 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-5103f6d9-6c38-4ce1-acef-1fce54a05193 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '2d30f26e52424a7e92a0099fbc480683'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:46:07.745365 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-5103f6d9-6c38-4ce1-acef-1fce54a05193 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '2d30f26e52424a7e92a0099fbc480683'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:46:07.752252 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-5103f6d9-6c38-4ce1-acef-1fce54a05193 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '2d30f26e52424a7e92a0099fbc480683'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:46:07.758993 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1446/2883] 10.222.0.50 () {58 vars in 1224 bytes} [Tue Sep 13 16:46:07 2022] DELETE /identity/v3/users/2d30f26e52424a7e92a0099fbc480683 => generated 0 bytes in 158 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 16:46:07.930643 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-564394a4-4f15-4705-9286-2008ece8daf3 admin admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:46:07.932233 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-564394a4-4f15-4705-9286-2008ece8daf3 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:46:07.946760 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-564394a4-4f15-4705-9286-2008ece8daf3 admin admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:46:08.005791 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cbf1fad-bf8f-4316-8233-3c6c489fd502 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:46:08.006562 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cbf1fad-bf8f-4316-8233-3c6c489fd502 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:46:08.006847 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cbf1fad-bf8f-4316-8233-3c6c489fd502 admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:46:08.007156 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cbf1fad-bf8f-4316-8233-3c6c489fd502 admin admin] PATH_INFO: `/v3/projects/a8cb6e56acdb45c6aafdd3024471e8eb` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:46:08.008321 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8cbf1fad-bf8f-4316-8233-3c6c489fd502 admin admin] RBAC: Authorizing `identity:delete_project(project_id=a8cb6e56acdb45c6aafdd3024471e8eb)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:46:08.010244 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8cbf1fad-bf8f-4316-8233-3c6c489fd502 admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:46:08.061985 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-8cbf1fad-bf8f-4316-8233-3c6c489fd502 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'a8cb6e56acdb45c6aafdd3024471e8eb', 'request_id': 'req-8cbf1fad-bf8f-4316-8233-3c6c489fd502'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:46:08.069969 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-8cbf1fad-bf8f-4316-8233-3c6c489fd502 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'a8cb6e56acdb45c6aafdd3024471e8eb', 'request_id': 'req-8cbf1fad-bf8f-4316-8233-3c6c489fd502'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:46:08.076153 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-8cbf1fad-bf8f-4316-8233-3c6c489fd502 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'a8cb6e56acdb45c6aafdd3024471e8eb', 'request_id': 'req-8cbf1fad-bf8f-4316-8233-3c6c489fd502'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:46:08.088125 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-8cbf1fad-bf8f-4316-8233-3c6c489fd502 admin admin] The token cache is being invalidate because project a8cb6e56acdb45c6aafdd3024471e8eb was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81438) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 13 16:46:08.088504 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-8cbf1fad-bf8f-4316-8233-3c6c489fd502 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:46:08.089288 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-8cbf1fad-bf8f-4316-8233-3c6c489fd502 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:46:08.090864 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1438/2884] 10.222.0.50 () {58 vars in 1233 bytes} [Tue Sep 13 16:46:07 2022] DELETE /identity/v3/projects/a8cb6e56acdb45c6aafdd3024471e8eb => generated 0 bytes in 161 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 16:46:09.948750 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5103f6d9-6c38-4ce1-acef-1fce54a05193 admin admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:46:09.950156 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-5103f6d9-6c38-4ce1-acef-1fce54a05193 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:46:09.963879 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5103f6d9-6c38-4ce1-acef-1fce54a05193 admin admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:46:10.019744 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87c6ec7c-678a-4df6-bca8-941c81376e0e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:46:10.020365 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87c6ec7c-678a-4df6-bca8-941c81376e0e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:46:10.020951 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87c6ec7c-678a-4df6-bca8-941c81376e0e admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:46:10.021254 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87c6ec7c-678a-4df6-bca8-941c81376e0e admin admin] PATH_INFO: `/v3/users/1e1b4b76596c4c7aa9c1a7e26c055772` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:46:10.022376 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87c6ec7c-678a-4df6-bca8-941c81376e0e admin admin] RBAC: Authorizing `identity:delete_user(user_id=1e1b4b76596c4c7aa9c1a7e26c055772)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:46:10.024364 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87c6ec7c-678a-4df6-bca8-941c81376e0e admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:46:10.057312 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=84,keystone:DELETE=15,keystone:INSERT=1 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:46:10.073071 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-87c6ec7c-678a-4df6-bca8-941c81376e0e admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '1e1b4b76596c4c7aa9c1a7e26c055772'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:46:10.080055 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-87c6ec7c-678a-4df6-bca8-941c81376e0e admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '1e1b4b76596c4c7aa9c1a7e26c055772'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:46:10.092349 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-87c6ec7c-678a-4df6-bca8-941c81376e0e admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '1e1b4b76596c4c7aa9c1a7e26c055772'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:46:10.093034 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-87c6ec7c-678a-4df6-bca8-941c81376e0e admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '1e1b4b76596c4c7aa9c1a7e26c055772'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:46:10.099860 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-87c6ec7c-678a-4df6-bca8-941c81376e0e admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '1e1b4b76596c4c7aa9c1a7e26c055772'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:46:10.105685 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1447/2885] 10.222.0.50 () {58 vars in 1224 bytes} [Tue Sep 13 16:46:09 2022] DELETE /identity/v3/users/1e1b4b76596c4c7aa9c1a7e26c055772 => generated 0 bytes in 157 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 16:46:10.110232 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a7ba457-2aba-42eb-b1b0-9694581509ec None None] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:46:10.110683 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a7ba457-2aba-42eb-b1b0-9694581509ec None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:46:10.110791 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a7ba457-2aba-42eb-b1b0-9694581509ec None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:46:10.130800 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.auth.core [None req-1a7ba457-2aba-42eb-b1b0-9694581509ec None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81438) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:46:10.132046 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-1a7ba457-2aba-42eb-b1b0-9694581509ec None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:46:10.181736 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=87,keystone:DELETE=11,keystone:INSERT=2 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:46:10.193675 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1439/2886] 10.222.0.50 () {58 vars in 958 bytes} [Tue Sep 13 16:46:10 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:46:10.203317 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87c6ec7c-678a-4df6-bca8-941c81376e0e admin admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:46:10.204737 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-87c6ec7c-678a-4df6-bca8-941c81376e0e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:46:10.261381 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87c6ec7c-678a-4df6-bca8-941c81376e0e admin admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:46:10.264992 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33c05f61-f7bb-45aa-a7e2-1720aaec9b72 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f436f07939d4855a21b195bc7566244', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:46:10.265621 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33c05f61-f7bb-45aa-a7e2-1720aaec9b72 service neutron] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:46:10.265900 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33c05f61-f7bb-45aa-a7e2-1720aaec9b72 service neutron] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:46:10.266174 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33c05f61-f7bb-45aa-a7e2-1720aaec9b72 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:46:10.275430 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33c05f61-f7bb-45aa-a7e2-1720aaec9b72 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:46:10.276852 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33c05f61-f7bb-45aa-a7e2-1720aaec9b72 service neutron] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:46:10.286107 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1448/2887] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 16:46:10 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 83 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 16:46:10.544478 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a7ba457-2aba-42eb-b1b0-9694581509ec None None] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:46:10.545915 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-1a7ba457-2aba-42eb-b1b0-9694581509ec None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:46:10.560799 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a7ba457-2aba-42eb-b1b0-9694581509ec None None] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:46:10.566021 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b1ee636-5a61-4f09-9f75-3f612e88cdb3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:46:10.566858 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b1ee636-5a61-4f09-9f75-3f612e88cdb3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:46:10.567178 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b1ee636-5a61-4f09-9f75-3f612e88cdb3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:46:10.568077 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b1ee636-5a61-4f09-9f75-3f612e88cdb3 admin admin] PATH_INFO: `/v3/projects/a1fcf85aa12a42c99c427cc2dc9476e2` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:46:10.569233 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b1ee636-5a61-4f09-9f75-3f612e88cdb3 admin admin] RBAC: Authorizing `identity:delete_project(project_id=a1fcf85aa12a42c99c427cc2dc9476e2)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:46:10.571307 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b1ee636-5a61-4f09-9f75-3f612e88cdb3 admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:46:10.618137 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-0b1ee636-5a61-4f09-9f75-3f612e88cdb3 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'a1fcf85aa12a42c99c427cc2dc9476e2', 'request_id': 'req-0b1ee636-5a61-4f09-9f75-3f612e88cdb3'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:46:10.625464 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-0b1ee636-5a61-4f09-9f75-3f612e88cdb3 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'a1fcf85aa12a42c99c427cc2dc9476e2', 'request_id': 'req-0b1ee636-5a61-4f09-9f75-3f612e88cdb3'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:46:10.631709 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-0b1ee636-5a61-4f09-9f75-3f612e88cdb3 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'a1fcf85aa12a42c99c427cc2dc9476e2', 'request_id': 'req-0b1ee636-5a61-4f09-9f75-3f612e88cdb3'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:46:10.642757 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-0b1ee636-5a61-4f09-9f75-3f612e88cdb3 admin admin] The token cache is being invalidate because project a1fcf85aa12a42c99c427cc2dc9476e2 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81438) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 13 16:46:10.643086 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-0b1ee636-5a61-4f09-9f75-3f612e88cdb3 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:46:10.643719 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-0b1ee636-5a61-4f09-9f75-3f612e88cdb3 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:46:10.645182 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1440/2888] 10.222.0.50 () {58 vars in 1233 bytes} [Tue Sep 13 16:46:10 2022] DELETE /identity/v3/projects/a1fcf85aa12a42c99c427cc2dc9476e2 => generated 0 bytes in 101 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 16:46:10.649723 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33c05f61-f7bb-45aa-a7e2-1720aaec9b72 service neutron] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:46:10.659271 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33c05f61-f7bb-45aa-a7e2-1720aaec9b72 service neutron] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:46:10.715937 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-396e932c-7ef5-490a-a15a-faf34a8b92d7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:46:10.716561 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-396e932c-7ef5-490a-a15a-faf34a8b92d7 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:46:10.716843 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-396e932c-7ef5-490a-a15a-faf34a8b92d7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:46:10.717146 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-396e932c-7ef5-490a-a15a-faf34a8b92d7 admin admin] PATH_INFO: `/v3/users/b2f304bdba5c4bb8922be96ce5695bde` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:46:10.718267 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-396e932c-7ef5-490a-a15a-faf34a8b92d7 admin admin] RBAC: Authorizing `identity:delete_user(user_id=b2f304bdba5c4bb8922be96ce5695bde)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:46:10.720276 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-396e932c-7ef5-490a-a15a-faf34a8b92d7 admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:46:10.765770 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-396e932c-7ef5-490a-a15a-faf34a8b92d7 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'b2f304bdba5c4bb8922be96ce5695bde'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:46:10.769290 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=85,keystone:DELETE=13,keystone:INSERT=2 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:46:10.773698 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-396e932c-7ef5-490a-a15a-faf34a8b92d7 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'b2f304bdba5c4bb8922be96ce5695bde'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:46:10.787583 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-396e932c-7ef5-490a-a15a-faf34a8b92d7 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'b2f304bdba5c4bb8922be96ce5695bde'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:46:10.788211 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-396e932c-7ef5-490a-a15a-faf34a8b92d7 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'b2f304bdba5c4bb8922be96ce5695bde'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:46:10.794890 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-396e932c-7ef5-490a-a15a-faf34a8b92d7 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'b2f304bdba5c4bb8922be96ce5695bde'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:46:10.800637 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1449/2889] 10.222.0.50 () {58 vars in 1224 bytes} [Tue Sep 13 16:46:10 2022] DELETE /identity/v3/users/b2f304bdba5c4bb8922be96ce5695bde => generated 0 bytes in 151 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 16:46:11.078390 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b1ee636-5a61-4f09-9f75-3f612e88cdb3 admin admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:46:11.079917 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-0b1ee636-5a61-4f09-9f75-3f612e88cdb3 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:46:11.093391 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b1ee636-5a61-4f09-9f75-3f612e88cdb3 admin admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:46:11.152504 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a75318ec-f8d7-40b4-be1a-57b8aa0e8b00 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:46:11.153141 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a75318ec-f8d7-40b4-be1a-57b8aa0e8b00 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:46:11.153401 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a75318ec-f8d7-40b4-be1a-57b8aa0e8b00 admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:46:11.153672 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a75318ec-f8d7-40b4-be1a-57b8aa0e8b00 admin admin] PATH_INFO: `/v3/projects/c9acaa93110c4f119215036a7107f59a` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:46:11.154800 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a75318ec-f8d7-40b4-be1a-57b8aa0e8b00 admin admin] RBAC: Authorizing `identity:delete_project(project_id=c9acaa93110c4f119215036a7107f59a)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:46:11.156732 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a75318ec-f8d7-40b4-be1a-57b8aa0e8b00 admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:46:11.202688 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-a75318ec-f8d7-40b4-be1a-57b8aa0e8b00 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'c9acaa93110c4f119215036a7107f59a', 'request_id': 'req-a75318ec-f8d7-40b4-be1a-57b8aa0e8b00'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:46:11.211060 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-a75318ec-f8d7-40b4-be1a-57b8aa0e8b00 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'c9acaa93110c4f119215036a7107f59a', 'request_id': 'req-a75318ec-f8d7-40b4-be1a-57b8aa0e8b00'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:46:11.216741 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-a75318ec-f8d7-40b4-be1a-57b8aa0e8b00 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'c9acaa93110c4f119215036a7107f59a', 'request_id': 'req-a75318ec-f8d7-40b4-be1a-57b8aa0e8b00'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:46:11.228216 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-a75318ec-f8d7-40b4-be1a-57b8aa0e8b00 admin admin] The token cache is being invalidate because project c9acaa93110c4f119215036a7107f59a was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81438) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 13 16:46:11.228443 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-a75318ec-f8d7-40b4-be1a-57b8aa0e8b00 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:46:11.229085 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-a75318ec-f8d7-40b4-be1a-57b8aa0e8b00 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:46:11.230550 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1441/2890] 10.222.0.50 () {58 vars in 1233 bytes} [Tue Sep 13 16:46:11 2022] DELETE /identity/v3/projects/c9acaa93110c4f119215036a7107f59a => generated 0 bytes in 153 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 16:46:11.235395 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b08d2ea7-4cb6-4cb3-8c44-7ec333367c03 None None] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:46:11.235840 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b08d2ea7-4cb6-4cb3-8c44-7ec333367c03 None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:46:11.235929 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b08d2ea7-4cb6-4cb3-8c44-7ec333367c03 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:46:11.256334 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.auth.core [None req-b08d2ea7-4cb6-4cb3-8c44-7ec333367c03 None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81437) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:46:11.257539 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-b08d2ea7-4cb6-4cb3-8c44-7ec333367c03 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:46:11.315481 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1450/2891] 10.222.0.50 () {58 vars in 958 bytes} [Tue Sep 13 16:46:11 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 81 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:46:11.323206 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f36b48b2-7f43-4645-8319-b3523c604101 None None] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:46:11.323493 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f36b48b2-7f43-4645-8319-b3523c604101 None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:46:11.323771 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f36b48b2-7f43-4645-8319-b3523c604101 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:46:11.343388 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.auth.core [None req-f36b48b2-7f43-4645-8319-b3523c604101 None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81438) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:46:11.344589 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-f36b48b2-7f43-4645-8319-b3523c604101 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:46:11.353058 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1442/2892] 10.222.0.50 () {58 vars in 958 bytes} [Tue Sep 13 16:46:11 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 13 16:46:11.357169 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b08d2ea7-4cb6-4cb3-8c44-7ec333367c03 None None] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:46:11.367324 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b08d2ea7-4cb6-4cb3-8c44-7ec333367c03 None None] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:46:11.371911 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e50cbb5-0077-4647-a94a-bea0ddbd8d3a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:46:11.372514 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e50cbb5-0077-4647-a94a-bea0ddbd8d3a admin admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:46:11.372794 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e50cbb5-0077-4647-a94a-bea0ddbd8d3a admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:46:11.373069 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e50cbb5-0077-4647-a94a-bea0ddbd8d3a admin admin] PATH_INFO: `/v3/domains` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:46:11.373744 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e50cbb5-0077-4647-a94a-bea0ddbd8d3a admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81437) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 16:46:11.374052 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e50cbb5-0077-4647-a94a-bea0ddbd8d3a admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:46:11.375711 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e50cbb5-0077-4647-a94a-bea0ddbd8d3a admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:46:11.387204 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1451/2893] 10.222.0.50 () {58 vars in 1153 bytes} [Tue Sep 13 16:46:11 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 16:46:11.391427 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f36b48b2-7f43-4645-8319-b3523c604101 None None] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:46:11.397379 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=84,keystone:DELETE=14,keystone:INSERT=2 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:46:11.400946 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f36b48b2-7f43-4645-8319-b3523c604101 None None] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:46:11.407183 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-406a31a5-a0df-4368-9988-014f891563a5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:46:11.407942 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-406a31a5-a0df-4368-9988-014f891563a5 admin admin] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:46:11.408388 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-406a31a5-a0df-4368-9988-014f891563a5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:46:11.408873 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-406a31a5-a0df-4368-9988-014f891563a5 admin admin] PATH_INFO: `/v3/projects` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:46:11.409698 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-406a31a5-a0df-4368-9988-014f891563a5 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:46:11.412198 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-406a31a5-a0df-4368-9988-014f891563a5 admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:46:11.439849 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1443/2894] 10.222.0.50 () {60 vars in 1153 bytes} [Tue Sep 13 16:46:11 2022] POST /identity/v3/projects => generated 393 bytes in 49 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 16:46:11.444113 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e50cbb5-0077-4647-a94a-bea0ddbd8d3a admin admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:46:11.454272 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e50cbb5-0077-4647-a94a-bea0ddbd8d3a admin admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:46:11.511348 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9ed829f-c6c8-4537-9573-61531cf03550 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:46:11.511866 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9ed829f-c6c8-4537-9573-61531cf03550 admin admin] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:46:11.512144 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9ed829f-c6c8-4537-9573-61531cf03550 admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:46:11.512419 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9ed829f-c6c8-4537-9573-61531cf03550 admin admin] PATH_INFO: `/v3/users` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:46:11.513065 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9ed829f-c6c8-4537-9573-61531cf03550 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:46:11.514992 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9ed829f-c6c8-4537-9573-61531cf03550 admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:46:11.536158 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1452/2895] 10.222.0.50 () {60 vars in 1144 bytes} [Tue Sep 13 16:46:11 2022] POST /identity/v3/users => generated 297 bytes in 92 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 16:46:11.540844 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-406a31a5-a0df-4368-9988-014f891563a5 admin admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:46:11.551260 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-406a31a5-a0df-4368-9988-014f891563a5 admin admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:46:11.556469 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-426bcb6a-9fc4-4ca2-9efb-0368d2937532 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:46:11.557061 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-426bcb6a-9fc4-4ca2-9efb-0368d2937532 admin admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:46:11.557340 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-426bcb6a-9fc4-4ca2-9efb-0368d2937532 admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:46:11.557612 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-426bcb6a-9fc4-4ca2-9efb-0368d2937532 admin admin] PATH_INFO: `/v3/roles` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:46:11.558243 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-426bcb6a-9fc4-4ca2-9efb-0368d2937532 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:46:11.559858 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-426bcb6a-9fc4-4ca2-9efb-0368d2937532 admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:46:11.570107 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1444/2896] 10.222.0.50 () {58 vars in 1122 bytes} [Tue Sep 13 16:46:11 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 13 16:46:11.573887 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9ed829f-c6c8-4537-9573-61531cf03550 admin admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:46:11.584352 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9ed829f-c6c8-4537-9573-61531cf03550 admin admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:46:11.588965 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ce52e6c-ef0f-40a4-b757-7014edb16b62 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:46:11.589551 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ce52e6c-ef0f-40a4-b757-7014edb16b62 admin admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:46:11.589828 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ce52e6c-ef0f-40a4-b757-7014edb16b62 admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:46:11.590093 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ce52e6c-ef0f-40a4-b757-7014edb16b62 admin admin] PATH_INFO: `/v3/roles` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:46:11.590723 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ce52e6c-ef0f-40a4-b757-7014edb16b62 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:46:11.592384 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ce52e6c-ef0f-40a4-b757-7014edb16b62 admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:46:11.602782 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1453/2897] 10.222.0.50 () {58 vars in 1122 bytes} [Tue Sep 13 16:46:11 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 16:46:11.606647 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-426bcb6a-9fc4-4ca2-9efb-0368d2937532 admin admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:46:11.617316 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-426bcb6a-9fc4-4ca2-9efb-0368d2937532 admin admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:46:11.621935 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18194d83-6fb5-4795-944b-a64b936d43a8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:46:11.622577 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18194d83-6fb5-4795-944b-a64b936d43a8 admin admin] REQUEST_METHOD: `PUT` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:46:11.622857 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18194d83-6fb5-4795-944b-a64b936d43a8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:46:11.623172 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18194d83-6fb5-4795-944b-a64b936d43a8 admin admin] PATH_INFO: `/v3/projects/8cc7421739464b95885dd003beda5486/users/0522f925edf640918458c62db0618f40/roles/15db4aaca3264ae29bde38903ba43471` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:46:11.638226 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18194d83-6fb5-4795-944b-a64b936d43a8 admin admin] RBAC: Authorizing `identity:create_grant(project_id=8cc7421739464b95885dd003beda5486, user_id=0522f925edf640918458c62db0618f40, role_id=15db4aaca3264ae29bde38903ba43471)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:46:11.640972 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18194d83-6fb5-4795-944b-a64b936d43a8 admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:46:11.653647 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1445/2898] 10.222.0.50 () {60 vars in 1483 bytes} [Tue Sep 13 16:46:11 2022] PUT /identity/v3/projects/8cc7421739464b95885dd003beda5486/users/0522f925edf640918458c62db0618f40/roles/15db4aaca3264ae29bde38903ba43471 => generated 0 bytes in 47 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 13 16:46:11.664089 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7ba831f-6618-430e-a803-a62354d28c4b None None] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:46:11.664376 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7ba831f-6618-430e-a803-a62354d28c4b None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:46:11.664649 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7ba831f-6618-430e-a803-a62354d28c4b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:46:11.693302 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.auth.core [None req-f7ba831f-6618-430e-a803-a62354d28c4b None None] MFA Rules not processed for user `0522f925edf640918458c62db0618f40`. Rule list: `[]` (Enabled: `True`). {{(pid=81437) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:46:11.694555 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-f7ba831f-6618-430e-a803-a62354d28c4b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:46:11.718233 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=95,keystone:DELETE=2,keystone:INSERT=3 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:46:11.760740 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1454/2899] 10.222.0.50 () {58 vars in 958 bytes} [Tue Sep 13 16:46:11 2022] POST /identity/v3/auth/tokens => generated 3270 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:46:11.765437 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18194d83-6fb5-4795-944b-a64b936d43a8 admin admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:46:11.825748 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18194d83-6fb5-4795-944b-a64b936d43a8 admin admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:46:11.830590 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85046e47-0fde-44bd-85db-4ab3872a81f9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:46:11.831213 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85046e47-0fde-44bd-85db-4ab3872a81f9 admin admin] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:46:11.831491 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85046e47-0fde-44bd-85db-4ab3872a81f9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:46:11.831762 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85046e47-0fde-44bd-85db-4ab3872a81f9 admin admin] PATH_INFO: `/v3/projects` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:46:11.832374 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85046e47-0fde-44bd-85db-4ab3872a81f9 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:46:11.834292 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85046e47-0fde-44bd-85db-4ab3872a81f9 admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:46:11.859703 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1446/2900] 10.222.0.50 () {60 vars in 1153 bytes} [Tue Sep 13 16:46:11 2022] POST /identity/v3/projects => generated 393 bytes in 95 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 16:46:11.863756 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7ba831f-6618-430e-a803-a62354d28c4b None None] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:46:11.874814 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7ba831f-6618-430e-a803-a62354d28c4b None None] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:46:11.929380 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c23bc379-7c2d-4402-a3dc-4ffdc63f48b3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:46:11.929972 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c23bc379-7c2d-4402-a3dc-4ffdc63f48b3 admin admin] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:46:11.930252 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c23bc379-7c2d-4402-a3dc-4ffdc63f48b3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:46:11.930525 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c23bc379-7c2d-4402-a3dc-4ffdc63f48b3 admin admin] PATH_INFO: `/v3/users` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:46:11.931177 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c23bc379-7c2d-4402-a3dc-4ffdc63f48b3 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:46:11.933084 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c23bc379-7c2d-4402-a3dc-4ffdc63f48b3 admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:46:11.955919 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1455/2901] 10.222.0.50 () {60 vars in 1144 bytes} [Tue Sep 13 16:46:11 2022] POST /identity/v3/users => generated 303 bytes in 93 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 16:46:11.959965 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85046e47-0fde-44bd-85db-4ab3872a81f9 admin admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:46:11.970115 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85046e47-0fde-44bd-85db-4ab3872a81f9 admin admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:46:11.975396 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a452dbb7-1aef-412c-acef-76c8e02a1df1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:46:11.975985 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a452dbb7-1aef-412c-acef-76c8e02a1df1 admin admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:46:11.976261 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a452dbb7-1aef-412c-acef-76c8e02a1df1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:46:11.976533 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a452dbb7-1aef-412c-acef-76c8e02a1df1 admin admin] PATH_INFO: `/v3/roles` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:46:11.977161 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a452dbb7-1aef-412c-acef-76c8e02a1df1 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:46:11.978745 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a452dbb7-1aef-412c-acef-76c8e02a1df1 admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:46:11.989156 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1447/2902] 10.222.0.50 () {58 vars in 1122 bytes} [Tue Sep 13 16:46:11 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 16:46:11.993807 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c23bc379-7c2d-4402-a3dc-4ffdc63f48b3 admin admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:46:12.003902 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c23bc379-7c2d-4402-a3dc-4ffdc63f48b3 admin admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:46:12.008681 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7c0f88d-6379-4f21-8f03-f75bb846e473 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:46:12.009334 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7c0f88d-6379-4f21-8f03-f75bb846e473 admin admin] REQUEST_METHOD: `PUT` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:46:12.009617 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7c0f88d-6379-4f21-8f03-f75bb846e473 admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:46:12.009896 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7c0f88d-6379-4f21-8f03-f75bb846e473 admin admin] PATH_INFO: `/v3/projects/c405c2c28d554875bd4a55f0ddc9532d/users/4aac3b3790e9460e92541a90d54fad6e/roles/031d4e1a23e34f71858fc4a6dfbd4ccb` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:46:12.024602 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7c0f88d-6379-4f21-8f03-f75bb846e473 admin admin] RBAC: Authorizing `identity:create_grant(project_id=c405c2c28d554875bd4a55f0ddc9532d, user_id=4aac3b3790e9460e92541a90d54fad6e, role_id=031d4e1a23e34f71858fc4a6dfbd4ccb)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:46:12.027201 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7c0f88d-6379-4f21-8f03-f75bb846e473 admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:46:12.043957 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1456/2903] 10.222.0.50 () {60 vars in 1483 bytes} [Tue Sep 13 16:46:11 2022] PUT /identity/v3/projects/c405c2c28d554875bd4a55f0ddc9532d/users/4aac3b3790e9460e92541a90d54fad6e/roles/031d4e1a23e34f71858fc4a6dfbd4ccb => generated 0 bytes in 48 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 16:46:12.044995 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a452dbb7-1aef-412c-acef-76c8e02a1df1 admin admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:46:12.055819 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a452dbb7-1aef-412c-acef-76c8e02a1df1 admin admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:46:12.111103 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-faec5a51-a197-4657-b678-e25597fb2cc5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:46:12.111711 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faec5a51-a197-4657-b678-e25597fb2cc5 admin admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:46:12.111991 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faec5a51-a197-4657-b678-e25597fb2cc5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:46:12.112265 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faec5a51-a197-4657-b678-e25597fb2cc5 admin admin] PATH_INFO: `/v3/roles` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:46:12.112896 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-faec5a51-a197-4657-b678-e25597fb2cc5 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:46:12.115152 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-faec5a51-a197-4657-b678-e25597fb2cc5 admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:46:12.124714 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1448/2904] 10.222.0.50 () {58 vars in 1122 bytes} [Tue Sep 13 16:46:12 2022] GET /identity/v3/roles => generated 1420 bytes in 80 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 16:46:12.128789 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7c0f88d-6379-4f21-8f03-f75bb846e473 admin admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:46:12.139287 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7c0f88d-6379-4f21-8f03-f75bb846e473 admin admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:46:12.144054 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f940df55-0ece-4e77-ac7a-bd5d7d5beaef admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:46:12.144706 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f940df55-0ece-4e77-ac7a-bd5d7d5beaef admin admin] REQUEST_METHOD: `PUT` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:46:12.144983 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f940df55-0ece-4e77-ac7a-bd5d7d5beaef admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:46:12.145257 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f940df55-0ece-4e77-ac7a-bd5d7d5beaef admin admin] PATH_INFO: `/v3/projects/c405c2c28d554875bd4a55f0ddc9532d/users/4aac3b3790e9460e92541a90d54fad6e/roles/031d4e1a23e34f71858fc4a6dfbd4ccb` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:46:12.147743 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f940df55-0ece-4e77-ac7a-bd5d7d5beaef admin admin] RBAC: Authorizing `identity:create_grant(project_id=c405c2c28d554875bd4a55f0ddc9532d, user_id=4aac3b3790e9460e92541a90d54fad6e, role_id=031d4e1a23e34f71858fc4a6dfbd4ccb)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:46:12.150484 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f940df55-0ece-4e77-ac7a-bd5d7d5beaef admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:46:12.160598 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1457/2905] 10.222.0.50 () {60 vars in 1483 bytes} [Tue Sep 13 16:46:12 2022] PUT /identity/v3/projects/c405c2c28d554875bd4a55f0ddc9532d/users/4aac3b3790e9460e92541a90d54fad6e/roles/031d4e1a23e34f71858fc4a6dfbd4ccb => generated 0 bytes in 32 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 16:46:12.172683 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d647c7d6-6e9b-42fc-afc8-340b74f74608 None None] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:46:12.173152 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d647c7d6-6e9b-42fc-afc8-340b74f74608 None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:46:12.173242 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d647c7d6-6e9b-42fc-afc8-340b74f74608 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:46:12.201392 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.auth.core [None req-d647c7d6-6e9b-42fc-afc8-340b74f74608 None None] MFA Rules not processed for user `4aac3b3790e9460e92541a90d54fad6e`. Rule list: `[]` (Enabled: `True`). {{(pid=81438) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:46:12.202625 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-d647c7d6-6e9b-42fc-afc8-340b74f74608 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:46:12.207855 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:46:12.271743 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1449/2906] 10.222.0.50 () {58 vars in 958 bytes} [Tue Sep 13 16:46:12 2022] POST /identity/v3/auth/tokens => generated 3337 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:46:12.282336 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f940df55-0ece-4e77-ac7a-bd5d7d5beaef admin admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:46:12.284094 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-f940df55-0ece-4e77-ac7a-bd5d7d5beaef admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:46:12.340847 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f940df55-0ece-4e77-ac7a-bd5d7d5beaef admin admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:46:12.344481 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d51618a-0540-4451-b957-2f80f0019d6d service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7fd6b5afd67b48f9b0f0597ccd4cf4df', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:46:12.345076 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d51618a-0540-4451-b957-2f80f0019d6d service cinder] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:46:12.345355 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d51618a-0540-4451-b957-2f80f0019d6d service cinder] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:46:12.345629 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d51618a-0540-4451-b957-2f80f0019d6d service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:46:12.355242 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d51618a-0540-4451-b957-2f80f0019d6d service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:46:12.356665 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d51618a-0540-4451-b957-2f80f0019d6d service cinder] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:46:12.366142 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1458/2907] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 16:46:12 2022] GET /identity/v3/auth/tokens => generated 3337 bytes in 84 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 16:46:12.414544 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d647c7d6-6e9b-42fc-afc8-340b74f74608 None None] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:46:12.415990 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-d647c7d6-6e9b-42fc-afc8-340b74f74608 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:46:12.427943 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d647c7d6-6e9b-42fc-afc8-340b74f74608 None None] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:46:12.431550 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e4239ac-36ea-46ce-8f4b-1724257359f0 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7fd6b5afd67b48f9b0f0597ccd4cf4df', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:46:12.432205 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e4239ac-36ea-46ce-8f4b-1724257359f0 service cinder] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:46:12.432485 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e4239ac-36ea-46ce-8f4b-1724257359f0 service cinder] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:46:12.432759 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e4239ac-36ea-46ce-8f4b-1724257359f0 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:46:12.491171 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e4239ac-36ea-46ce-8f4b-1724257359f0 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:46:12.492598 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e4239ac-36ea-46ce-8f4b-1724257359f0 service cinder] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:46:12.501552 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1450/2908] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 16:46:12 2022] GET /identity/v3/auth/tokens => generated 3270 bytes in 88 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 13 16:46:15.879257 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d51618a-0540-4451-b957-2f80f0019d6d service cinder] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:46:15.899995 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:46:15.942926 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d51618a-0540-4451-b957-2f80f0019d6d service cinder] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:46:15.947437 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93016ac6-1285-4788-a5d8-f06285ea811c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:46:15.948063 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93016ac6-1285-4788-a5d8-f06285ea811c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:46:15.948341 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93016ac6-1285-4788-a5d8-f06285ea811c admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:46:15.948613 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93016ac6-1285-4788-a5d8-f06285ea811c admin admin] PATH_INFO: `/v3/users/0522f925edf640918458c62db0618f40` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:46:15.949754 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93016ac6-1285-4788-a5d8-f06285ea811c admin admin] RBAC: Authorizing `identity:delete_user(user_id=0522f925edf640918458c62db0618f40)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:46:15.951703 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93016ac6-1285-4788-a5d8-f06285ea811c admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:46:15.998695 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-93016ac6-1285-4788-a5d8-f06285ea811c admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '0522f925edf640918458c62db0618f40'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:46:16.006098 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-93016ac6-1285-4788-a5d8-f06285ea811c admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '0522f925edf640918458c62db0618f40'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:46:16.018196 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-93016ac6-1285-4788-a5d8-f06285ea811c admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '0522f925edf640918458c62db0618f40'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:46:16.018833 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-93016ac6-1285-4788-a5d8-f06285ea811c admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '0522f925edf640918458c62db0618f40'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:46:16.026294 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-93016ac6-1285-4788-a5d8-f06285ea811c admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '0522f925edf640918458c62db0618f40'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:46:16.032141 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1459/2909] 10.222.0.50 () {58 vars in 1224 bytes} [Tue Sep 13 16:46:15 2022] DELETE /identity/v3/users/0522f925edf640918458c62db0618f40 => generated 0 bytes in 153 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 16:46:16.037294 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c028b74-fc30-45a3-a6f7-ee5883cded4e None None] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:46:16.037763 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c028b74-fc30-45a3-a6f7-ee5883cded4e None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:46:16.038054 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c028b74-fc30-45a3-a6f7-ee5883cded4e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:46:16.057584 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.auth.core [None req-5c028b74-fc30-45a3-a6f7-ee5883cded4e None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81438) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:46:16.058780 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-5c028b74-fc30-45a3-a6f7-ee5883cded4e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:46:16.118096 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1451/2910] 10.222.0.50 () {58 vars in 958 bytes} [Tue Sep 13 16:46:16 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:46:16.127637 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93016ac6-1285-4788-a5d8-f06285ea811c admin admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:46:16.129033 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-93016ac6-1285-4788-a5d8-f06285ea811c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:46:16.186009 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93016ac6-1285-4788-a5d8-f06285ea811c admin admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:46:16.189770 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e028bed1-96ee-42d9-ba3f-de894a09caef service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f436f07939d4855a21b195bc7566244', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:46:16.190388 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e028bed1-96ee-42d9-ba3f-de894a09caef service neutron] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:46:16.190656 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e028bed1-96ee-42d9-ba3f-de894a09caef service neutron] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:46:16.190962 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e028bed1-96ee-42d9-ba3f-de894a09caef service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:46:16.200208 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e028bed1-96ee-42d9-ba3f-de894a09caef service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:46:16.201707 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e028bed1-96ee-42d9-ba3f-de894a09caef service neutron] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:46:16.212158 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1460/2911] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 16:46:16 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 85 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 16:46:16.477212 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c028b74-fc30-45a3-a6f7-ee5883cded4e None None] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:46:16.478665 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-5c028b74-fc30-45a3-a6f7-ee5883cded4e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:46:16.491793 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c028b74-fc30-45a3-a6f7-ee5883cded4e None None] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:46:16.496613 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef9b5f0e-9e60-4719-bc6c-8d78e4b4c203 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:46:16.497277 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef9b5f0e-9e60-4719-bc6c-8d78e4b4c203 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:46:16.497553 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef9b5f0e-9e60-4719-bc6c-8d78e4b4c203 admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:46:16.497824 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef9b5f0e-9e60-4719-bc6c-8d78e4b4c203 admin admin] PATH_INFO: `/v3/projects/8cc7421739464b95885dd003beda5486` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:46:16.499574 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef9b5f0e-9e60-4719-bc6c-8d78e4b4c203 admin admin] RBAC: Authorizing `identity:delete_project(project_id=8cc7421739464b95885dd003beda5486)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:46:16.502063 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef9b5f0e-9e60-4719-bc6c-8d78e4b4c203 admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:46:16.540660 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=97,keystone:DELETE=3 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:46:16.552033 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-ef9b5f0e-9e60-4719-bc6c-8d78e4b4c203 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '8cc7421739464b95885dd003beda5486', 'request_id': 'req-ef9b5f0e-9e60-4719-bc6c-8d78e4b4c203'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:46:16.561514 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-ef9b5f0e-9e60-4719-bc6c-8d78e4b4c203 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '8cc7421739464b95885dd003beda5486', 'request_id': 'req-ef9b5f0e-9e60-4719-bc6c-8d78e4b4c203'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:46:16.567967 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-ef9b5f0e-9e60-4719-bc6c-8d78e4b4c203 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '8cc7421739464b95885dd003beda5486', 'request_id': 'req-ef9b5f0e-9e60-4719-bc6c-8d78e4b4c203'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:46:16.578865 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-ef9b5f0e-9e60-4719-bc6c-8d78e4b4c203 admin admin] The token cache is being invalidate because project 8cc7421739464b95885dd003beda5486 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81438) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 13 16:46:16.579168 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-ef9b5f0e-9e60-4719-bc6c-8d78e4b4c203 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:46:16.579829 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-ef9b5f0e-9e60-4719-bc6c-8d78e4b4c203 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:46:16.581556 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1452/2912] 10.222.0.50 () {58 vars in 1233 bytes} [Tue Sep 13 16:46:16 2022] DELETE /identity/v3/projects/8cc7421739464b95885dd003beda5486 => generated 0 bytes in 105 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 16:46:16.585573 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e028bed1-96ee-42d9-ba3f-de894a09caef service neutron] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:46:16.595255 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e028bed1-96ee-42d9-ba3f-de894a09caef service neutron] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:46:16.635706 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:46:16.653312 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1d5e0d8-4948-4c42-b100-d3f9a57716dd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:46:16.653939 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1d5e0d8-4948-4c42-b100-d3f9a57716dd admin admin] REQUEST_METHOD: `DELETE` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:46:16.654219 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1d5e0d8-4948-4c42-b100-d3f9a57716dd admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:46:16.654489 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1d5e0d8-4948-4c42-b100-d3f9a57716dd admin admin] PATH_INFO: `/v3/users/4aac3b3790e9460e92541a90d54fad6e` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:46:16.655622 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1d5e0d8-4948-4c42-b100-d3f9a57716dd admin admin] RBAC: Authorizing `identity:delete_user(user_id=4aac3b3790e9460e92541a90d54fad6e)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:46:16.657562 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1d5e0d8-4948-4c42-b100-d3f9a57716dd admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:46:16.704322 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-e1d5e0d8-4948-4c42-b100-d3f9a57716dd admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '4aac3b3790e9460e92541a90d54fad6e'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:46:16.713307 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-e1d5e0d8-4948-4c42-b100-d3f9a57716dd admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '4aac3b3790e9460e92541a90d54fad6e'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:46:16.725900 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-e1d5e0d8-4948-4c42-b100-d3f9a57716dd admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '4aac3b3790e9460e92541a90d54fad6e'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:46:16.726542 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-e1d5e0d8-4948-4c42-b100-d3f9a57716dd admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '4aac3b3790e9460e92541a90d54fad6e'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:46:16.733400 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-e1d5e0d8-4948-4c42-b100-d3f9a57716dd admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '4aac3b3790e9460e92541a90d54fad6e'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:46:16.739509 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1461/2913] 10.222.0.50 () {58 vars in 1224 bytes} [Tue Sep 13 16:46:16 2022] DELETE /identity/v3/users/4aac3b3790e9460e92541a90d54fad6e => generated 0 bytes in 154 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 16:46:17.013223 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef9b5f0e-9e60-4719-bc6c-8d78e4b4c203 admin admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:46:17.014802 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-ef9b5f0e-9e60-4719-bc6c-8d78e4b4c203 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:46:17.029139 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef9b5f0e-9e60-4719-bc6c-8d78e4b4c203 admin admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:46:17.096119 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40e0816b-b6aa-4de9-b596-bd33b9962942 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:46:17.096748 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40e0816b-b6aa-4de9-b596-bd33b9962942 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:46:17.097026 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40e0816b-b6aa-4de9-b596-bd33b9962942 admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:46:17.097298 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40e0816b-b6aa-4de9-b596-bd33b9962942 admin admin] PATH_INFO: `/v3/projects/c405c2c28d554875bd4a55f0ddc9532d` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:46:17.098471 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40e0816b-b6aa-4de9-b596-bd33b9962942 admin admin] RBAC: Authorizing `identity:delete_project(project_id=c405c2c28d554875bd4a55f0ddc9532d)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:46:17.100528 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40e0816b-b6aa-4de9-b596-bd33b9962942 admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:46:17.146143 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-40e0816b-b6aa-4de9-b596-bd33b9962942 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'c405c2c28d554875bd4a55f0ddc9532d', 'request_id': 'req-40e0816b-b6aa-4de9-b596-bd33b9962942'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:46:17.153313 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-40e0816b-b6aa-4de9-b596-bd33b9962942 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'c405c2c28d554875bd4a55f0ddc9532d', 'request_id': 'req-40e0816b-b6aa-4de9-b596-bd33b9962942'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:46:17.158877 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-40e0816b-b6aa-4de9-b596-bd33b9962942 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'c405c2c28d554875bd4a55f0ddc9532d', 'request_id': 'req-40e0816b-b6aa-4de9-b596-bd33b9962942'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:46:17.170186 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-40e0816b-b6aa-4de9-b596-bd33b9962942 admin admin] The token cache is being invalidate because project c405c2c28d554875bd4a55f0ddc9532d was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81438) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 13 16:46:17.170476 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-40e0816b-b6aa-4de9-b596-bd33b9962942 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:46:17.171141 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-40e0816b-b6aa-4de9-b596-bd33b9962942 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:46:17.172622 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1453/2914] 10.222.0.50 () {58 vars in 1233 bytes} [Tue Sep 13 16:46:17 2022] DELETE /identity/v3/projects/c405c2c28d554875bd4a55f0ddc9532d => generated 0 bytes in 160 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 16:46:17.179065 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a456d0a-daae-45dc-a729-6f47fd3844ac None None] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:46:17.179540 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a456d0a-daae-45dc-a729-6f47fd3844ac None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:46:17.179633 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a456d0a-daae-45dc-a729-6f47fd3844ac None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:46:17.199219 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.auth.core [None req-3a456d0a-daae-45dc-a729-6f47fd3844ac None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81437) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:46:17.200409 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-3a456d0a-daae-45dc-a729-6f47fd3844ac None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:46:17.258804 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1462/2915] 10.222.0.50 () {58 vars in 958 bytes} [Tue Sep 13 16:46:17 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 82 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:46:17.266497 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-069e7a97-ddda-4561-a500-b95b52b3696b None None] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:46:17.266856 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-069e7a97-ddda-4561-a500-b95b52b3696b None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:46:17.267244 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-069e7a97-ddda-4561-a500-b95b52b3696b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:46:17.287989 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.auth.core [None req-069e7a97-ddda-4561-a500-b95b52b3696b None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81438) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:46:17.289205 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-069e7a97-ddda-4561-a500-b95b52b3696b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:46:17.297466 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1454/2916] 10.222.0.50 () {58 vars in 958 bytes} [Tue Sep 13 16:46:17 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:46:17.302278 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a456d0a-daae-45dc-a729-6f47fd3844ac None None] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:46:17.312462 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a456d0a-daae-45dc-a729-6f47fd3844ac None None] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:46:17.317080 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ada702a6-2835-4279-975d-87533dd1a489 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:46:17.318014 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ada702a6-2835-4279-975d-87533dd1a489 admin admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:46:17.318327 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ada702a6-2835-4279-975d-87533dd1a489 admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:46:17.318597 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ada702a6-2835-4279-975d-87533dd1a489 admin admin] PATH_INFO: `/v3/domains` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:46:17.319272 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ada702a6-2835-4279-975d-87533dd1a489 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81437) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 16:46:17.319574 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ada702a6-2835-4279-975d-87533dd1a489 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:46:17.321200 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ada702a6-2835-4279-975d-87533dd1a489 admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:46:17.333024 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1463/2917] 10.222.0.50 () {58 vars in 1153 bytes} [Tue Sep 13 16:46:17 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 16:46:17.337460 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-069e7a97-ddda-4561-a500-b95b52b3696b None None] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:46:17.346632 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-069e7a97-ddda-4561-a500-b95b52b3696b None None] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:46:17.351822 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87b430e7-90a8-4bad-9855-a8beb5d5d9ad admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:46:17.352415 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87b430e7-90a8-4bad-9855-a8beb5d5d9ad admin admin] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:46:17.352712 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87b430e7-90a8-4bad-9855-a8beb5d5d9ad admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:46:17.352972 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87b430e7-90a8-4bad-9855-a8beb5d5d9ad admin admin] PATH_INFO: `/v3/projects` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:46:17.353594 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87b430e7-90a8-4bad-9855-a8beb5d5d9ad admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:46:17.355550 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87b430e7-90a8-4bad-9855-a8beb5d5d9ad admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:46:17.382523 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1455/2918] 10.222.0.50 () {60 vars in 1153 bytes} [Tue Sep 13 16:46:17 2022] POST /identity/v3/projects => generated 375 bytes in 45 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 16:46:17.386614 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ada702a6-2835-4279-975d-87533dd1a489 admin admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:46:17.396560 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ada702a6-2835-4279-975d-87533dd1a489 admin admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:46:17.440942 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:46:17.455706 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-169d8b30-d60c-4600-8d02-a75d1b66ca08 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:46:17.456318 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-169d8b30-d60c-4600-8d02-a75d1b66ca08 admin admin] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:46:17.456597 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-169d8b30-d60c-4600-8d02-a75d1b66ca08 admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:46:17.456872 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-169d8b30-d60c-4600-8d02-a75d1b66ca08 admin admin] PATH_INFO: `/v3/users` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:46:17.457505 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-169d8b30-d60c-4600-8d02-a75d1b66ca08 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:46:17.460099 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-169d8b30-d60c-4600-8d02-a75d1b66ca08 admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:46:17.481167 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1464/2919] 10.222.0.50 () {60 vars in 1144 bytes} [Tue Sep 13 16:46:17 2022] POST /identity/v3/users => generated 288 bytes in 95 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 16:46:17.485260 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87b430e7-90a8-4bad-9855-a8beb5d5d9ad admin admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:46:17.495418 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87b430e7-90a8-4bad-9855-a8beb5d5d9ad admin admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:46:17.500002 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-007a581e-3999-48e6-8d4b-fb277405dbcd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:46:17.500592 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-007a581e-3999-48e6-8d4b-fb277405dbcd admin admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:46:17.500872 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-007a581e-3999-48e6-8d4b-fb277405dbcd admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:46:17.501147 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-007a581e-3999-48e6-8d4b-fb277405dbcd admin admin] PATH_INFO: `/v3/roles` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:46:17.501780 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-007a581e-3999-48e6-8d4b-fb277405dbcd admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:46:17.503411 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-007a581e-3999-48e6-8d4b-fb277405dbcd admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:46:17.513143 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1456/2920] 10.222.0.50 () {58 vars in 1122 bytes} [Tue Sep 13 16:46:17 2022] GET /identity/v3/roles => generated 1420 bytes in 28 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 13 16:46:17.516946 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-169d8b30-d60c-4600-8d02-a75d1b66ca08 admin admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:46:17.527309 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-169d8b30-d60c-4600-8d02-a75d1b66ca08 admin admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:46:17.531910 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0beabd9-0624-46dc-9b43-43e305189c4f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:46:17.532496 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0beabd9-0624-46dc-9b43-43e305189c4f admin admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:46:17.532775 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0beabd9-0624-46dc-9b43-43e305189c4f admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:46:17.533051 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0beabd9-0624-46dc-9b43-43e305189c4f admin admin] PATH_INFO: `/v3/roles` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:46:17.533683 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0beabd9-0624-46dc-9b43-43e305189c4f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:46:17.535350 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0beabd9-0624-46dc-9b43-43e305189c4f admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:46:17.546868 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1465/2921] 10.222.0.50 () {58 vars in 1122 bytes} [Tue Sep 13 16:46:17 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 16:46:17.550839 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-007a581e-3999-48e6-8d4b-fb277405dbcd admin admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:46:17.562010 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-007a581e-3999-48e6-8d4b-fb277405dbcd admin admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:46:17.566624 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc3718b6-ff7f-4449-9327-cffe9b8f4d9a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:46:17.568166 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc3718b6-ff7f-4449-9327-cffe9b8f4d9a admin admin] REQUEST_METHOD: `PUT` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:46:17.568449 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc3718b6-ff7f-4449-9327-cffe9b8f4d9a admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:46:17.568722 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc3718b6-ff7f-4449-9327-cffe9b8f4d9a admin admin] PATH_INFO: `/v3/projects/5630176f88f84abbb46d01c117e8061d/users/184feeed996740bb82a59ff1ba9cba17/roles/15db4aaca3264ae29bde38903ba43471` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:46:17.580718 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=86,keystone:DELETE=11,keystone:INSERT=3 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:46:17.585432 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc3718b6-ff7f-4449-9327-cffe9b8f4d9a admin admin] RBAC: Authorizing `identity:create_grant(project_id=5630176f88f84abbb46d01c117e8061d, user_id=184feeed996740bb82a59ff1ba9cba17, role_id=15db4aaca3264ae29bde38903ba43471)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:46:17.588749 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc3718b6-ff7f-4449-9327-cffe9b8f4d9a admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:46:17.609349 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1457/2922] 10.222.0.50 () {60 vars in 1483 bytes} [Tue Sep 13 16:46:17 2022] PUT /identity/v3/projects/5630176f88f84abbb46d01c117e8061d/users/184feeed996740bb82a59ff1ba9cba17/roles/15db4aaca3264ae29bde38903ba43471 => generated 0 bytes in 59 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 16:46:17.621630 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54edbb05-4ef9-41df-9a97-e7f53f46e7ab None None] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:46:17.621918 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54edbb05-4ef9-41df-9a97-e7f53f46e7ab None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:46:17.622194 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54edbb05-4ef9-41df-9a97-e7f53f46e7ab None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:46:17.650700 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.auth.core [None req-54edbb05-4ef9-41df-9a97-e7f53f46e7ab None None] MFA Rules not processed for user `184feeed996740bb82a59ff1ba9cba17`. Rule list: `[]` (Enabled: `True`). {{(pid=81437) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:46:17.651942 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-54edbb05-4ef9-41df-9a97-e7f53f46e7ab None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:46:17.714936 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1466/2923] 10.222.0.50 () {58 vars in 958 bytes} [Tue Sep 13 16:46:17 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:46:17.719394 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc3718b6-ff7f-4449-9327-cffe9b8f4d9a admin admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:46:17.780328 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc3718b6-ff7f-4449-9327-cffe9b8f4d9a admin admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:46:17.784925 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-455619d6-b377-4040-8764-fcb3ae03e4bc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:46:17.785525 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-455619d6-b377-4040-8764-fcb3ae03e4bc admin admin] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:46:17.785804 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-455619d6-b377-4040-8764-fcb3ae03e4bc admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:46:17.786659 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-455619d6-b377-4040-8764-fcb3ae03e4bc admin admin] PATH_INFO: `/v3/projects` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:46:17.787429 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-455619d6-b377-4040-8764-fcb3ae03e4bc admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:46:17.789361 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-455619d6-b377-4040-8764-fcb3ae03e4bc admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:46:17.816936 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1458/2924] 10.222.0.50 () {60 vars in 1153 bytes} [Tue Sep 13 16:46:17 2022] POST /identity/v3/projects => generated 375 bytes in 98 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 16:46:17.821130 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54edbb05-4ef9-41df-9a97-e7f53f46e7ab None None] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:46:17.831840 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54edbb05-4ef9-41df-9a97-e7f53f46e7ab None None] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:46:17.888100 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43669a80-14c6-4439-ac5f-356d37517cff admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:46:17.888632 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43669a80-14c6-4439-ac5f-356d37517cff admin admin] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:46:17.888910 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43669a80-14c6-4439-ac5f-356d37517cff admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:46:17.889185 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43669a80-14c6-4439-ac5f-356d37517cff admin admin] PATH_INFO: `/v3/users` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:46:17.889836 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43669a80-14c6-4439-ac5f-356d37517cff admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:46:17.891915 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43669a80-14c6-4439-ac5f-356d37517cff admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:46:17.916393 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1467/2925] 10.222.0.50 () {60 vars in 1144 bytes} [Tue Sep 13 16:46:17 2022] POST /identity/v3/users => generated 294 bytes in 93 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 16:46:17.917556 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-455619d6-b377-4040-8764-fcb3ae03e4bc admin admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:46:17.928503 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-455619d6-b377-4040-8764-fcb3ae03e4bc admin admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:46:17.933092 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a964e77e-2457-4631-a128-bb2a1bca67f0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:46:17.933682 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a964e77e-2457-4631-a128-bb2a1bca67f0 admin admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:46:17.933963 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a964e77e-2457-4631-a128-bb2a1bca67f0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:46:17.934234 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a964e77e-2457-4631-a128-bb2a1bca67f0 admin admin] PATH_INFO: `/v3/roles` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:46:17.934864 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a964e77e-2457-4631-a128-bb2a1bca67f0 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:46:17.936478 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a964e77e-2457-4631-a128-bb2a1bca67f0 admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:46:17.947067 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1459/2926] 10.222.0.50 () {58 vars in 1122 bytes} [Tue Sep 13 16:46:17 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 13 16:46:17.951510 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43669a80-14c6-4439-ac5f-356d37517cff admin admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:46:17.962058 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43669a80-14c6-4439-ac5f-356d37517cff admin admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:46:17.967041 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d1a4ee2-3bc2-4a84-ba15-4264524e2a49 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:46:17.967692 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d1a4ee2-3bc2-4a84-ba15-4264524e2a49 admin admin] REQUEST_METHOD: `PUT` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:46:17.967973 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d1a4ee2-3bc2-4a84-ba15-4264524e2a49 admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:46:17.968245 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d1a4ee2-3bc2-4a84-ba15-4264524e2a49 admin admin] PATH_INFO: `/v3/projects/e20e60312d394bd98f247fd9bd38462d/users/5dbb0a76dd124548b3539e74f4312f32/roles/031d4e1a23e34f71858fc4a6dfbd4ccb` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:46:17.983635 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d1a4ee2-3bc2-4a84-ba15-4264524e2a49 admin admin] RBAC: Authorizing `identity:create_grant(project_id=e20e60312d394bd98f247fd9bd38462d, user_id=5dbb0a76dd124548b3539e74f4312f32, role_id=031d4e1a23e34f71858fc4a6dfbd4ccb)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:46:17.986377 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d1a4ee2-3bc2-4a84-ba15-4264524e2a49 admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:46:17.999459 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1468/2927] 10.222.0.50 () {60 vars in 1483 bytes} [Tue Sep 13 16:46:17 2022] PUT /identity/v3/projects/e20e60312d394bd98f247fd9bd38462d/users/5dbb0a76dd124548b3539e74f4312f32/roles/031d4e1a23e34f71858fc4a6dfbd4ccb => generated 0 bytes in 49 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 13 16:46:18.003559 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a964e77e-2457-4631-a128-bb2a1bca67f0 admin admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:46:18.013869 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a964e77e-2457-4631-a128-bb2a1bca67f0 admin admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:46:18.072291 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c4d53f2-5a7d-4429-ad31-16739a1a1710 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:46:18.072920 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c4d53f2-5a7d-4429-ad31-16739a1a1710 admin admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:46:18.073196 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c4d53f2-5a7d-4429-ad31-16739a1a1710 admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:46:18.073463 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c4d53f2-5a7d-4429-ad31-16739a1a1710 admin admin] PATH_INFO: `/v3/roles` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:46:18.074084 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c4d53f2-5a7d-4429-ad31-16739a1a1710 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:46:18.075692 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c4d53f2-5a7d-4429-ad31-16739a1a1710 admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:46:18.085979 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1460/2928] 10.222.0.50 () {58 vars in 1122 bytes} [Tue Sep 13 16:46:18 2022] GET /identity/v3/roles => generated 1420 bytes in 83 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 13 16:46:18.090279 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d1a4ee2-3bc2-4a84-ba15-4264524e2a49 admin admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:46:18.101365 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d1a4ee2-3bc2-4a84-ba15-4264524e2a49 admin admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:46:18.105990 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30876e02-0c8e-48ff-884d-9a09e2a78776 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:46:18.106632 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30876e02-0c8e-48ff-884d-9a09e2a78776 admin admin] REQUEST_METHOD: `PUT` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:46:18.106931 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30876e02-0c8e-48ff-884d-9a09e2a78776 admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:46:18.107206 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30876e02-0c8e-48ff-884d-9a09e2a78776 admin admin] PATH_INFO: `/v3/projects/e20e60312d394bd98f247fd9bd38462d/users/5dbb0a76dd124548b3539e74f4312f32/roles/031d4e1a23e34f71858fc4a6dfbd4ccb` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:46:18.109078 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30876e02-0c8e-48ff-884d-9a09e2a78776 admin admin] RBAC: Authorizing `identity:create_grant(project_id=e20e60312d394bd98f247fd9bd38462d, user_id=5dbb0a76dd124548b3539e74f4312f32, role_id=031d4e1a23e34f71858fc4a6dfbd4ccb)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:46:18.111738 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30876e02-0c8e-48ff-884d-9a09e2a78776 admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:46:18.122807 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1469/2929] 10.222.0.50 () {60 vars in 1483 bytes} [Tue Sep 13 16:46:18 2022] PUT /identity/v3/projects/e20e60312d394bd98f247fd9bd38462d/users/5dbb0a76dd124548b3539e74f4312f32/roles/031d4e1a23e34f71858fc4a6dfbd4ccb => generated 0 bytes in 33 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 16:46:18.133665 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b83212a-687d-42ae-b018-e827efaeacb5 None None] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:46:18.134178 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b83212a-687d-42ae-b018-e827efaeacb5 None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:46:18.134284 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b83212a-687d-42ae-b018-e827efaeacb5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:46:18.163465 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.auth.core [None req-9b83212a-687d-42ae-b018-e827efaeacb5 None None] MFA Rules not processed for user `5dbb0a76dd124548b3539e74f4312f32`. Rule list: `[]` (Enabled: `True`). {{(pid=81438) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:46:18.164600 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-9b83212a-687d-42ae-b018-e827efaeacb5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:46:18.212849 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:46:18.234891 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1461/2930] 10.222.0.50 () {58 vars in 958 bytes} [Tue Sep 13 16:46:18 2022] POST /identity/v3/auth/tokens => generated 3319 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:46:18.245797 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30876e02-0c8e-48ff-884d-9a09e2a78776 admin admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:46:18.247545 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-30876e02-0c8e-48ff-884d-9a09e2a78776 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:46:18.258446 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=92,keystone:INSERT=8 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:46:18.306119 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30876e02-0c8e-48ff-884d-9a09e2a78776 admin admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:46:18.309689 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2dc698fc-a2f9-43c8-acb4-a21b979d7a29 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7fd6b5afd67b48f9b0f0597ccd4cf4df', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:46:18.310281 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dc698fc-a2f9-43c8-acb4-a21b979d7a29 service cinder] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:46:18.310556 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dc698fc-a2f9-43c8-acb4-a21b979d7a29 service cinder] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:46:18.310825 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dc698fc-a2f9-43c8-acb4-a21b979d7a29 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:46:18.366106 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2dc698fc-a2f9-43c8-acb4-a21b979d7a29 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:46:18.367565 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2dc698fc-a2f9-43c8-acb4-a21b979d7a29 service cinder] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:46:18.376479 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1470/2931] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 16:46:18 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 131 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 16:46:20.907046 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b83212a-687d-42ae-b018-e827efaeacb5 None None] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:46:20.971501 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b83212a-687d-42ae-b018-e827efaeacb5 None None] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:46:20.976551 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8d60d07-a973-451b-9486-821ce84163be admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:46:20.977167 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8d60d07-a973-451b-9486-821ce84163be admin admin] REQUEST_METHOD: `DELETE` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:46:20.977443 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8d60d07-a973-451b-9486-821ce84163be admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:46:20.977712 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8d60d07-a973-451b-9486-821ce84163be admin admin] PATH_INFO: `/v3/users/184feeed996740bb82a59ff1ba9cba17` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:46:20.978822 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8d60d07-a973-451b-9486-821ce84163be admin admin] RBAC: Authorizing `identity:delete_user(user_id=184feeed996740bb82a59ff1ba9cba17)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:46:20.980745 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8d60d07-a973-451b-9486-821ce84163be admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:46:21.028276 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-c8d60d07-a973-451b-9486-821ce84163be admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '184feeed996740bb82a59ff1ba9cba17'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:46:21.036331 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-c8d60d07-a973-451b-9486-821ce84163be admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '184feeed996740bb82a59ff1ba9cba17'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:46:21.048502 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-c8d60d07-a973-451b-9486-821ce84163be admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '184feeed996740bb82a59ff1ba9cba17'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:46:21.049224 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-c8d60d07-a973-451b-9486-821ce84163be admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '184feeed996740bb82a59ff1ba9cba17'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:46:21.056270 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-c8d60d07-a973-451b-9486-821ce84163be admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '184feeed996740bb82a59ff1ba9cba17'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:46:21.062210 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1462/2932] 10.222.0.50 () {58 vars in 1224 bytes} [Tue Sep 13 16:46:20 2022] DELETE /identity/v3/users/184feeed996740bb82a59ff1ba9cba17 => generated 0 bytes in 156 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 16:46:21.066974 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed424085-4a95-4ea0-9504-baa5a915f019 None None] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:46:21.068069 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed424085-4a95-4ea0-9504-baa5a915f019 None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:46:21.068069 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed424085-4a95-4ea0-9504-baa5a915f019 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:46:21.088574 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.auth.core [None req-ed424085-4a95-4ea0-9504-baa5a915f019 None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81437) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:46:21.089778 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-ed424085-4a95-4ea0-9504-baa5a915f019 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:46:21.147478 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1471/2933] 10.222.0.50 () {58 vars in 958 bytes} [Tue Sep 13 16:46:21 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 81 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:46:21.156936 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8d60d07-a973-451b-9486-821ce84163be admin admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:46:21.158348 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-c8d60d07-a973-451b-9486-821ce84163be admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:46:21.216220 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8d60d07-a973-451b-9486-821ce84163be admin admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:46:21.219824 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57ece956-c385-46f2-add5-7b0b4ef31da2 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f436f07939d4855a21b195bc7566244', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:46:21.220420 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57ece956-c385-46f2-add5-7b0b4ef31da2 service neutron] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:46:21.220696 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57ece956-c385-46f2-add5-7b0b4ef31da2 service neutron] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:46:21.220965 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57ece956-c385-46f2-add5-7b0b4ef31da2 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:46:21.229904 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57ece956-c385-46f2-add5-7b0b4ef31da2 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:46:21.231414 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57ece956-c385-46f2-add5-7b0b4ef31da2 service neutron] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:46:21.241252 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1463/2934] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 16:46:21 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 85 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 16:46:21.509343 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed424085-4a95-4ea0-9504-baa5a915f019 None None] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:46:21.510810 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-ed424085-4a95-4ea0-9504-baa5a915f019 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:46:21.523979 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed424085-4a95-4ea0-9504-baa5a915f019 None None] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:46:21.528998 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-994d9986-bef9-4c12-b767-cdcd4ac65ab4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:46:21.529640 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-994d9986-bef9-4c12-b767-cdcd4ac65ab4 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:46:21.529919 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-994d9986-bef9-4c12-b767-cdcd4ac65ab4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:46:21.530190 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-994d9986-bef9-4c12-b767-cdcd4ac65ab4 admin admin] PATH_INFO: `/v3/projects/5630176f88f84abbb46d01c117e8061d` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:46:21.531463 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-994d9986-bef9-4c12-b767-cdcd4ac65ab4 admin admin] RBAC: Authorizing `identity:delete_project(project_id=5630176f88f84abbb46d01c117e8061d)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:46:21.533381 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-994d9986-bef9-4c12-b767-cdcd4ac65ab4 admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:46:21.582787 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-994d9986-bef9-4c12-b767-cdcd4ac65ab4 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '5630176f88f84abbb46d01c117e8061d', 'request_id': 'req-994d9986-bef9-4c12-b767-cdcd4ac65ab4'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:46:21.590702 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-994d9986-bef9-4c12-b767-cdcd4ac65ab4 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '5630176f88f84abbb46d01c117e8061d', 'request_id': 'req-994d9986-bef9-4c12-b767-cdcd4ac65ab4'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:46:21.592794 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=94,keystone:DELETE=5,keystone:INSERT=1 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:46:21.597535 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-994d9986-bef9-4c12-b767-cdcd4ac65ab4 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '5630176f88f84abbb46d01c117e8061d', 'request_id': 'req-994d9986-bef9-4c12-b767-cdcd4ac65ab4'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:46:21.609954 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-994d9986-bef9-4c12-b767-cdcd4ac65ab4 admin admin] The token cache is being invalidate because project 5630176f88f84abbb46d01c117e8061d was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81437) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 13 16:46:21.610245 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-994d9986-bef9-4c12-b767-cdcd4ac65ab4 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:46:21.610886 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-994d9986-bef9-4c12-b767-cdcd4ac65ab4 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:46:21.612405 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1472/2935] 10.222.0.50 () {58 vars in 1233 bytes} [Tue Sep 13 16:46:21 2022] DELETE /identity/v3/projects/5630176f88f84abbb46d01c117e8061d => generated 0 bytes in 104 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 16:46:21.617114 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57ece956-c385-46f2-add5-7b0b4ef31da2 service neutron] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:46:21.626560 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57ece956-c385-46f2-add5-7b0b4ef31da2 service neutron] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:46:21.631140 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:46:21.684563 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd45621b-aaef-4fc1-8d5f-679abcbeb8a1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:46:21.685113 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd45621b-aaef-4fc1-8d5f-679abcbeb8a1 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:46:21.685392 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd45621b-aaef-4fc1-8d5f-679abcbeb8a1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:46:21.685692 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd45621b-aaef-4fc1-8d5f-679abcbeb8a1 admin admin] PATH_INFO: `/v3/users/5dbb0a76dd124548b3539e74f4312f32` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:46:21.686794 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd45621b-aaef-4fc1-8d5f-679abcbeb8a1 admin admin] RBAC: Authorizing `identity:delete_user(user_id=5dbb0a76dd124548b3539e74f4312f32)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:46:21.688763 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd45621b-aaef-4fc1-8d5f-679abcbeb8a1 admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:46:21.733172 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-bd45621b-aaef-4fc1-8d5f-679abcbeb8a1 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '5dbb0a76dd124548b3539e74f4312f32'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:46:21.740633 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-bd45621b-aaef-4fc1-8d5f-679abcbeb8a1 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '5dbb0a76dd124548b3539e74f4312f32'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:46:21.752470 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-bd45621b-aaef-4fc1-8d5f-679abcbeb8a1 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '5dbb0a76dd124548b3539e74f4312f32'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:46:21.753109 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-bd45621b-aaef-4fc1-8d5f-679abcbeb8a1 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '5dbb0a76dd124548b3539e74f4312f32'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:46:21.760000 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-bd45621b-aaef-4fc1-8d5f-679abcbeb8a1 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '5dbb0a76dd124548b3539e74f4312f32'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:46:21.765950 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1464/2936] 10.222.0.50 () {58 vars in 1224 bytes} [Tue Sep 13 16:46:21 2022] DELETE /identity/v3/users/5dbb0a76dd124548b3539e74f4312f32 => generated 0 bytes in 149 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 16:46:22.039146 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-994d9986-bef9-4c12-b767-cdcd4ac65ab4 admin admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:46:22.040691 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-994d9986-bef9-4c12-b767-cdcd4ac65ab4 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:46:22.056493 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-994d9986-bef9-4c12-b767-cdcd4ac65ab4 admin admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:46:22.116462 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbdfb0de-ec01-4724-a6ab-0126d7deb242 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:46:22.117121 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbdfb0de-ec01-4724-a6ab-0126d7deb242 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:46:22.117361 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbdfb0de-ec01-4724-a6ab-0126d7deb242 admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:46:22.117635 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbdfb0de-ec01-4724-a6ab-0126d7deb242 admin admin] PATH_INFO: `/v3/projects/e20e60312d394bd98f247fd9bd38462d` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:46:22.118748 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbdfb0de-ec01-4724-a6ab-0126d7deb242 admin admin] RBAC: Authorizing `identity:delete_project(project_id=e20e60312d394bd98f247fd9bd38462d)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:46:22.120747 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbdfb0de-ec01-4724-a6ab-0126d7deb242 admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:46:22.166645 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-bbdfb0de-ec01-4724-a6ab-0126d7deb242 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'e20e60312d394bd98f247fd9bd38462d', 'request_id': 'req-bbdfb0de-ec01-4724-a6ab-0126d7deb242'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:46:22.173572 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-bbdfb0de-ec01-4724-a6ab-0126d7deb242 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'e20e60312d394bd98f247fd9bd38462d', 'request_id': 'req-bbdfb0de-ec01-4724-a6ab-0126d7deb242'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:46:22.179682 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-bbdfb0de-ec01-4724-a6ab-0126d7deb242 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'e20e60312d394bd98f247fd9bd38462d', 'request_id': 'req-bbdfb0de-ec01-4724-a6ab-0126d7deb242'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:46:22.189950 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-bbdfb0de-ec01-4724-a6ab-0126d7deb242 admin admin] The token cache is being invalidate because project e20e60312d394bd98f247fd9bd38462d was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81437) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 13 16:46:22.190233 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-bbdfb0de-ec01-4724-a6ab-0126d7deb242 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:46:22.190861 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-bbdfb0de-ec01-4724-a6ab-0126d7deb242 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:46:22.192431 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1473/2937] 10.222.0.50 () {58 vars in 1233 bytes} [Tue Sep 13 16:46:22 2022] DELETE /identity/v3/projects/e20e60312d394bd98f247fd9bd38462d => generated 0 bytes in 154 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 16:46:22.199556 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec5e1be1-c771-426c-b775-8d5e89c0754f None None] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:46:22.200035 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec5e1be1-c771-426c-b775-8d5e89c0754f None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:46:22.200147 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec5e1be1-c771-426c-b775-8d5e89c0754f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:46:22.220276 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.auth.core [None req-ec5e1be1-c771-426c-b775-8d5e89c0754f None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81438) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:46:22.221388 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-ec5e1be1-c771-426c-b775-8d5e89c0754f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:46:22.279044 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1465/2938] 10.222.0.50 () {58 vars in 958 bytes} [Tue Sep 13 16:46:22 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 80 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:46:22.287651 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-809ffb5f-5ad4-4b8e-b110-ed0f9fb59c57 None None] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:46:22.287938 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-809ffb5f-5ad4-4b8e-b110-ed0f9fb59c57 None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:46:22.288211 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-809ffb5f-5ad4-4b8e-b110-ed0f9fb59c57 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:46:22.307645 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.auth.core [None req-809ffb5f-5ad4-4b8e-b110-ed0f9fb59c57 None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81437) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:46:22.308824 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-809ffb5f-5ad4-4b8e-b110-ed0f9fb59c57 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:46:22.316593 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1474/2939] 10.222.0.50 () {58 vars in 958 bytes} [Tue Sep 13 16:46:22 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:46:22.321274 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec5e1be1-c771-426c-b775-8d5e89c0754f None None] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:46:22.331565 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec5e1be1-c771-426c-b775-8d5e89c0754f None None] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:46:22.336793 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3d8b4b9-1469-4cdf-9de1-868e5ff6b58f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:46:22.337424 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3d8b4b9-1469-4cdf-9de1-868e5ff6b58f admin admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:46:22.337700 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3d8b4b9-1469-4cdf-9de1-868e5ff6b58f admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:46:22.337974 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3d8b4b9-1469-4cdf-9de1-868e5ff6b58f admin admin] PATH_INFO: `/v3/domains` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:46:22.338616 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3d8b4b9-1469-4cdf-9de1-868e5ff6b58f admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81438) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 16:46:22.338941 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3d8b4b9-1469-4cdf-9de1-868e5ff6b58f admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:46:22.340527 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3d8b4b9-1469-4cdf-9de1-868e5ff6b58f admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:46:22.351845 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1466/2940] 10.222.0.50 () {58 vars in 1153 bytes} [Tue Sep 13 16:46:22 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 16:46:22.356096 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-809ffb5f-5ad4-4b8e-b110-ed0f9fb59c57 None None] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:46:22.365271 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-809ffb5f-5ad4-4b8e-b110-ed0f9fb59c57 None None] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:46:22.369814 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d220327-98cb-42ed-9961-68ad7ef4a41c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:46:22.370397 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d220327-98cb-42ed-9961-68ad7ef4a41c admin admin] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:46:22.370670 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d220327-98cb-42ed-9961-68ad7ef4a41c admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:46:22.370968 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d220327-98cb-42ed-9961-68ad7ef4a41c admin admin] PATH_INFO: `/v3/projects` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:46:22.371592 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d220327-98cb-42ed-9961-68ad7ef4a41c admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:46:22.373483 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d220327-98cb-42ed-9961-68ad7ef4a41c admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:46:22.401768 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1475/2941] 10.222.0.50 () {60 vars in 1153 bytes} [Tue Sep 13 16:46:22 2022] POST /identity/v3/projects => generated 371 bytes in 46 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 16:46:22.406001 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3d8b4b9-1469-4cdf-9de1-868e5ff6b58f admin admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:46:22.416078 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3d8b4b9-1469-4cdf-9de1-868e5ff6b58f admin admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:46:22.427746 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:46:22.474260 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bab3a43-9b05-4e44-883f-9f0bed10f0be admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:46:22.474792 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bab3a43-9b05-4e44-883f-9f0bed10f0be admin admin] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:46:22.475103 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bab3a43-9b05-4e44-883f-9f0bed10f0be admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:46:22.475375 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bab3a43-9b05-4e44-883f-9f0bed10f0be admin admin] PATH_INFO: `/v3/users` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:46:22.476005 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bab3a43-9b05-4e44-883f-9f0bed10f0be admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:46:22.477899 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bab3a43-9b05-4e44-883f-9f0bed10f0be admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:46:22.498800 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1467/2942] 10.222.0.50 () {60 vars in 1144 bytes} [Tue Sep 13 16:46:22 2022] POST /identity/v3/users => generated 286 bytes in 93 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 16:46:22.502455 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d220327-98cb-42ed-9961-68ad7ef4a41c admin admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:46:22.513317 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d220327-98cb-42ed-9961-68ad7ef4a41c admin admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:46:22.517935 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ca0a76f-9f4c-455c-993c-c2bf97ae92ec admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:46:22.518526 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ca0a76f-9f4c-455c-993c-c2bf97ae92ec admin admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:46:22.518804 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ca0a76f-9f4c-455c-993c-c2bf97ae92ec admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:46:22.519114 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ca0a76f-9f4c-455c-993c-c2bf97ae92ec admin admin] PATH_INFO: `/v3/roles` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:46:22.519758 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ca0a76f-9f4c-455c-993c-c2bf97ae92ec admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:46:22.522105 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ca0a76f-9f4c-455c-993c-c2bf97ae92ec admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:46:22.531973 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1476/2943] 10.222.0.50 () {58 vars in 1122 bytes} [Tue Sep 13 16:46:22 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 16:46:22.535883 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bab3a43-9b05-4e44-883f-9f0bed10f0be admin admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:46:22.546760 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bab3a43-9b05-4e44-883f-9f0bed10f0be admin admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:46:22.551468 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab1de452-5476-4d36-959b-613241488571 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:46:22.552092 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab1de452-5476-4d36-959b-613241488571 admin admin] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:46:22.552375 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab1de452-5476-4d36-959b-613241488571 admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:46:22.552651 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab1de452-5476-4d36-959b-613241488571 admin admin] PATH_INFO: `/v3/roles` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:46:22.553293 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab1de452-5476-4d36-959b-613241488571 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:46:22.554887 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab1de452-5476-4d36-959b-613241488571 admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:46:22.565519 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1468/2944] 10.222.0.50 () {58 vars in 1122 bytes} [Tue Sep 13 16:46:22 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 16:46:22.570476 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ca0a76f-9f4c-455c-993c-c2bf97ae92ec admin admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:46:22.580797 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ca0a76f-9f4c-455c-993c-c2bf97ae92ec admin admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:46:22.586183 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d170115-4cf6-4eca-b795-350e7b5e805a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:46:22.586855 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d170115-4cf6-4eca-b795-350e7b5e805a admin admin] REQUEST_METHOD: `PUT` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:46:22.587197 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d170115-4cf6-4eca-b795-350e7b5e805a admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:46:22.587477 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d170115-4cf6-4eca-b795-350e7b5e805a admin admin] PATH_INFO: `/v3/projects/971ad6e8cfa54529a3e6aac596203515/users/0e8f8e2bf7f440f291b47dfd9357bc14/roles/15db4aaca3264ae29bde38903ba43471` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:46:22.602027 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d170115-4cf6-4eca-b795-350e7b5e805a admin admin] RBAC: Authorizing `identity:create_grant(project_id=971ad6e8cfa54529a3e6aac596203515, user_id=0e8f8e2bf7f440f291b47dfd9357bc14, role_id=15db4aaca3264ae29bde38903ba43471)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:46:22.604743 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d170115-4cf6-4eca-b795-350e7b5e805a admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:46:22.618576 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1477/2945] 10.222.0.50 () {60 vars in 1483 bytes} [Tue Sep 13 16:46:22 2022] PUT /identity/v3/projects/971ad6e8cfa54529a3e6aac596203515/users/0e8f8e2bf7f440f291b47dfd9357bc14/roles/15db4aaca3264ae29bde38903ba43471 => generated 0 bytes in 49 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 16:46:22.623681 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-550a450e-7db1-40b2-8867-8968c45fcaaf None None] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:46:22.623973 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-550a450e-7db1-40b2-8867-8968c45fcaaf None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:46:22.624252 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-550a450e-7db1-40b2-8867-8968c45fcaaf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:46:22.643398 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.auth.core [None req-550a450e-7db1-40b2-8867-8968c45fcaaf None None] MFA Rules not processed for user `e0a915a0eb3c42c7af97dcc7eed5b42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81438) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:46:22.644587 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-550a450e-7db1-40b2-8867-8968c45fcaaf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:46:22.702963 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1469/2946] 10.222.0.50 () {58 vars in 958 bytes} [Tue Sep 13 16:46:22 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 80 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:46:22.713113 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d170115-4cf6-4eca-b795-350e7b5e805a admin admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:46:22.714824 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-6d170115-4cf6-4eca-b795-350e7b5e805a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:46:22.729000 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=88,keystone:DELETE=9,keystone:INSERT=3 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:46:22.773768 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d170115-4cf6-4eca-b795-350e7b5e805a admin admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:46:22.777380 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb08b6a6-0dd3-48be-b44d-bd2ad618d7be service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f436f07939d4855a21b195bc7566244', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:46:22.777973 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb08b6a6-0dd3-48be-b44d-bd2ad618d7be service neutron] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:46:22.778254 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb08b6a6-0dd3-48be-b44d-bd2ad618d7be service neutron] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:46:22.778520 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb08b6a6-0dd3-48be-b44d-bd2ad618d7be service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:46:22.787266 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb08b6a6-0dd3-48be-b44d-bd2ad618d7be service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:46:22.788822 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb08b6a6-0dd3-48be-b44d-bd2ad618d7be service neutron] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:46:22.798098 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1478/2947] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 16:46:22 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 16:46:26.759979 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ce3f047-6e9b-4931-af62-000ea9937d8f None None] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:46:26.760506 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ce3f047-6e9b-4931-af62-000ea9937d8f None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:46:26.760605 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ce3f047-6e9b-4931-af62-000ea9937d8f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:46:26.790929 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.auth.core [None req-8ce3f047-6e9b-4931-af62-000ea9937d8f None None] MFA Rules not processed for user `0e8f8e2bf7f440f291b47dfd9357bc14`. Rule list: `[]` (Enabled: `True`). {{(pid=81438) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:46:26.792171 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-8ce3f047-6e9b-4931-af62-000ea9937d8f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:46:26.862152 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1470/2948] 10.222.0.50 () {58 vars in 958 bytes} [Tue Sep 13 16:46:26 2022] POST /identity/v3/auth/tokens => generated 3248 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:46:26.867224 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb08b6a6-0dd3-48be-b44d-bd2ad618d7be service neutron] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:46:26.877243 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb08b6a6-0dd3-48be-b44d-bd2ad618d7be service neutron] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:46:26.882471 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e2965eb-4635-4760-842f-a7a86b6c5072 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:46:26.883107 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e2965eb-4635-4760-842f-a7a86b6c5072 admin admin] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:46:26.883389 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e2965eb-4635-4760-842f-a7a86b6c5072 admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:46:26.883664 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e2965eb-4635-4760-842f-a7a86b6c5072 admin admin] PATH_INFO: `/v3/projects` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:46:26.884309 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e2965eb-4635-4760-842f-a7a86b6c5072 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:46:26.886232 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e2965eb-4635-4760-842f-a7a86b6c5072 admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:46:26.914533 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1479/2949] 10.222.0.50 () {60 vars in 1153 bytes} [Tue Sep 13 16:46:26 2022] POST /identity/v3/projects => generated 371 bytes in 48 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 16:46:26.918601 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ce3f047-6e9b-4931-af62-000ea9937d8f None None] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:46:26.929937 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ce3f047-6e9b-4931-af62-000ea9937d8f None None] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:46:26.951578 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:46:26.992760 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12facc3f-105a-4681-8083-702f7adc6f59 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:46:26.993478 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12facc3f-105a-4681-8083-702f7adc6f59 admin admin] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:46:26.993828 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12facc3f-105a-4681-8083-702f7adc6f59 admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:46:26.994169 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12facc3f-105a-4681-8083-702f7adc6f59 admin admin] PATH_INFO: `/v3/users` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:46:26.994815 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12facc3f-105a-4681-8083-702f7adc6f59 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:46:26.996751 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12facc3f-105a-4681-8083-702f7adc6f59 admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:46:27.018512 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1471/2950] 10.222.0.50 () {60 vars in 1144 bytes} [Tue Sep 13 16:46:26 2022] POST /identity/v3/users => generated 292 bytes in 100 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 16:46:27.023230 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e2965eb-4635-4760-842f-a7a86b6c5072 admin admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:46:27.033417 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e2965eb-4635-4760-842f-a7a86b6c5072 admin admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:46:27.038138 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab98cb44-b664-4dd8-8f73-b13b0d4aebb0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:46:27.038671 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab98cb44-b664-4dd8-8f73-b13b0d4aebb0 admin admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:46:27.038985 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab98cb44-b664-4dd8-8f73-b13b0d4aebb0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:46:27.039259 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab98cb44-b664-4dd8-8f73-b13b0d4aebb0 admin admin] PATH_INFO: `/v3/roles` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:46:27.039901 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab98cb44-b664-4dd8-8f73-b13b0d4aebb0 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:46:27.041504 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab98cb44-b664-4dd8-8f73-b13b0d4aebb0 admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:46:27.052389 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1480/2951] 10.222.0.50 () {58 vars in 1122 bytes} [Tue Sep 13 16:46:27 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 16:46:27.056505 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12facc3f-105a-4681-8083-702f7adc6f59 admin admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:46:27.067467 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12facc3f-105a-4681-8083-702f7adc6f59 admin admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:46:27.072288 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f49e14b-8cab-4f2f-9725-0b6ae6fb8a03 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:46:27.072945 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f49e14b-8cab-4f2f-9725-0b6ae6fb8a03 admin admin] REQUEST_METHOD: `PUT` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:46:27.073228 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f49e14b-8cab-4f2f-9725-0b6ae6fb8a03 admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:46:27.073505 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f49e14b-8cab-4f2f-9725-0b6ae6fb8a03 admin admin] PATH_INFO: `/v3/projects/2e5f6804a4f4475d8580763fe4029ea5/users/f586ba17a8e24707bfc02790060a2a4c/roles/031d4e1a23e34f71858fc4a6dfbd4ccb` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:46:27.089087 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f49e14b-8cab-4f2f-9725-0b6ae6fb8a03 admin admin] RBAC: Authorizing `identity:create_grant(project_id=2e5f6804a4f4475d8580763fe4029ea5, user_id=f586ba17a8e24707bfc02790060a2a4c, role_id=031d4e1a23e34f71858fc4a6dfbd4ccb)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:46:27.091761 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f49e14b-8cab-4f2f-9725-0b6ae6fb8a03 admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:46:27.104344 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1472/2952] 10.222.0.50 () {60 vars in 1483 bytes} [Tue Sep 13 16:46:27 2022] PUT /identity/v3/projects/2e5f6804a4f4475d8580763fe4029ea5/users/f586ba17a8e24707bfc02790060a2a4c/roles/031d4e1a23e34f71858fc4a6dfbd4ccb => generated 0 bytes in 48 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 16:46:27.108362 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab98cb44-b664-4dd8-8f73-b13b0d4aebb0 admin admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:46:27.118633 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab98cb44-b664-4dd8-8f73-b13b0d4aebb0 admin admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:46:27.177093 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a72dddb7-21d9-42d0-b78a-85a6240a3e8c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:46:27.177705 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a72dddb7-21d9-42d0-b78a-85a6240a3e8c admin admin] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:46:27.178010 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a72dddb7-21d9-42d0-b78a-85a6240a3e8c admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:46:27.178288 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a72dddb7-21d9-42d0-b78a-85a6240a3e8c admin admin] PATH_INFO: `/v3/roles` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:46:27.178962 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a72dddb7-21d9-42d0-b78a-85a6240a3e8c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:46:27.180559 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a72dddb7-21d9-42d0-b78a-85a6240a3e8c admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:46:27.190270 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1481/2953] 10.222.0.50 () {58 vars in 1122 bytes} [Tue Sep 13 16:46:27 2022] GET /identity/v3/roles => generated 1420 bytes in 82 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 16:46:27.195084 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f49e14b-8cab-4f2f-9725-0b6ae6fb8a03 admin admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:46:27.205150 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f49e14b-8cab-4f2f-9725-0b6ae6fb8a03 admin admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:46:27.210213 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6afec66-ddac-4429-a933-51ee18b5d0b7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:46:27.210852 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6afec66-ddac-4429-a933-51ee18b5d0b7 admin admin] REQUEST_METHOD: `PUT` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:46:27.211163 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6afec66-ddac-4429-a933-51ee18b5d0b7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:46:27.211434 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6afec66-ddac-4429-a933-51ee18b5d0b7 admin admin] PATH_INFO: `/v3/projects/2e5f6804a4f4475d8580763fe4029ea5/users/f586ba17a8e24707bfc02790060a2a4c/roles/031d4e1a23e34f71858fc4a6dfbd4ccb` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:46:27.213370 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6afec66-ddac-4429-a933-51ee18b5d0b7 admin admin] RBAC: Authorizing `identity:create_grant(project_id=2e5f6804a4f4475d8580763fe4029ea5, user_id=f586ba17a8e24707bfc02790060a2a4c, role_id=031d4e1a23e34f71858fc4a6dfbd4ccb)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:46:27.215993 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6afec66-ddac-4429-a933-51ee18b5d0b7 admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:46:27.226265 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1473/2954] 10.222.0.50 () {60 vars in 1483 bytes} [Tue Sep 13 16:46:27 2022] PUT /identity/v3/projects/2e5f6804a4f4475d8580763fe4029ea5/users/f586ba17a8e24707bfc02790060a2a4c/roles/031d4e1a23e34f71858fc4a6dfbd4ccb => generated 0 bytes in 32 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 16:46:31.663106 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e022ff9b-dcc0-4a18-89da-e2496fb3759c None None] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:46:31.663582 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e022ff9b-dcc0-4a18-89da-e2496fb3759c None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:46:31.663683 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e022ff9b-dcc0-4a18-89da-e2496fb3759c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:46:31.732516 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.auth.core [None req-e022ff9b-dcc0-4a18-89da-e2496fb3759c None None] MFA Rules not processed for user `f586ba17a8e24707bfc02790060a2a4c`. Rule list: `[]` (Enabled: `True`). {{(pid=81437) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:46:31.733133 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-e022ff9b-dcc0-4a18-89da-e2496fb3759c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:46:31.844175 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1482/2955] 10.222.0.50 () {58 vars in 958 bytes} [Tue Sep 13 16:46:31 2022] POST /identity/v3/auth/tokens => generated 3315 bytes in 183 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:46:31.856220 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6afec66-ddac-4429-a933-51ee18b5d0b7 admin admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:46:31.858223 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-e6afec66-ddac-4429-a933-51ee18b5d0b7 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:46:31.951124 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6afec66-ddac-4429-a933-51ee18b5d0b7 admin admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:46:31.960425 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a02fe2cd-7e42-40cb-9410-37d348c75e0e service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4e039b46c5e3445bbb39904b00c5430e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'admin', 'ResellerAdmin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:46:31.961072 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a02fe2cd-7e42-40cb-9410-37d348c75e0e service nova] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:46:31.961482 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a02fe2cd-7e42-40cb-9410-37d348c75e0e service nova] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:46:31.961653 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a02fe2cd-7e42-40cb-9410-37d348c75e0e service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:46:32.031982 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a02fe2cd-7e42-40cb-9410-37d348c75e0e service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:46:32.033605 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a02fe2cd-7e42-40cb-9410-37d348c75e0e service nova] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:46:32.050998 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1474/2956] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 16:46:31 2022] GET /identity/v3/auth/tokens => generated 3248 bytes in 195 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 16:46:33.120830 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc60b25b-0a24-4927-b4bb-e6c1dee65214 None None] REQUEST_METHOD: `POST` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:46:33.121290 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc60b25b-0a24-4927-b4bb-e6c1dee65214 None None] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:46:33.121383 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc60b25b-0a24-4927-b4bb-e6c1dee65214 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:46:33.129453 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:46:33.143591 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.auth.core [None req-cc60b25b-0a24-4927-b4bb-e6c1dee65214 None None] MFA Rules not processed for user `0e8f8e2bf7f440f291b47dfd9357bc14`. Rule list: `[]` (Enabled: `True`). {{(pid=81437) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:46:33.145256 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-cc60b25b-0a24-4927-b4bb-e6c1dee65214 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:46:33.153302 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1483/2957] 10.222.0.50 () {58 vars in 958 bytes} [Tue Sep 13 16:46:33 2022] POST /identity/v3/auth/tokens => generated 3248 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:46:42.040870 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=89,keystone:INSERT=5 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:46:43.140868 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=1 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:46:45.452097 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cc14041-68a8-4a8a-b7d3-d31e6489e223 None None] REQUEST_METHOD: `POST` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:46:45.452597 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cc14041-68a8-4a8a-b7d3-d31e6489e223 None None] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:46:45.452717 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cc14041-68a8-4a8a-b7d3-d31e6489e223 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:46:45.471825 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.auth.core [None req-2cc14041-68a8-4a8a-b7d3-d31e6489e223 None None] MFA Rules not processed for user `b6417e189b0d467480627cfd38331a9f`. Rule list: `[]` (Enabled: `True`). {{(pid=81438) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 16:46:45.472937 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-2cc14041-68a8-4a8a-b7d3-d31e6489e223 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:46:45.527473 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1475/2958] 10.222.0.50 () {58 vars in 958 bytes} [Tue Sep 13 16:46:45 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 76 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 16:46:53.096481 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc60b25b-0a24-4927-b4bb-e6c1dee65214 None None] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:46:53.125956 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc60b25b-0a24-4927-b4bb-e6c1dee65214 None None] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:46:53.137227 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdfba1be-62be-42ad-9d92-4b5908864d57 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4e039b46c5e3445bbb39904b00c5430e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'admin', 'ResellerAdmin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:46:53.138565 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdfba1be-62be-42ad-9d92-4b5908864d57 service nova] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:46:53.139314 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdfba1be-62be-42ad-9d92-4b5908864d57 service nova] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:46:53.140241 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdfba1be-62be-42ad-9d92-4b5908864d57 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:46:53.163062 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-cdfba1be-62be-42ad-9d92-4b5908864d57 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:46:53.194014 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cdfba1be-62be-42ad-9d92-4b5908864d57 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:46:53.198137 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cdfba1be-62be-42ad-9d92-4b5908864d57 service nova] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:46:53.222929 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1484/2959] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 16:46:53 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 129 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 16:46:55.520847 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=25 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:46:56.537773 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cc14041-68a8-4a8a-b7d3-d31e6489e223 None None] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:46:56.592285 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cc14041-68a8-4a8a-b7d3-d31e6489e223 None None] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:46:56.595830 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d361efa-08e6-4c0a-82b6-3dcae6aca0f7 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f436f07939d4855a21b195bc7566244', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:46:56.596437 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d361efa-08e6-4c0a-82b6-3dcae6aca0f7 service neutron] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:46:56.596715 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d361efa-08e6-4c0a-82b6-3dcae6aca0f7 service neutron] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:46:56.596991 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d361efa-08e6-4c0a-82b6-3dcae6aca0f7 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:46:56.599786 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-6d361efa-08e6-4c0a-82b6-3dcae6aca0f7 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:46:56.610581 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d361efa-08e6-4c0a-82b6-3dcae6aca0f7 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:46:56.612035 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d361efa-08e6-4c0a-82b6-3dcae6aca0f7 service neutron] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:46:56.621092 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1476/2960] 10.222.0.50 () {60 vars in 1396 bytes} [Tue Sep 13 16:46:56 2022] GET /identity/v3/auth/tokens?allow_expired=1 => generated 3254 bytes in 84 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 16:47:03.212197 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=6 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:47:05.980837 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdfba1be-62be-42ad-9d92-4b5908864d57 service nova] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:47:05.990924 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdfba1be-62be-42ad-9d92-4b5908864d57 service nova] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:47:05.994580 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92faeceb-283a-4438-8ea6-ca3672907775 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f436f07939d4855a21b195bc7566244', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:47:05.995200 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92faeceb-283a-4438-8ea6-ca3672907775 service neutron] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:47:05.995478 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92faeceb-283a-4438-8ea6-ca3672907775 service neutron] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:47:05.995749 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92faeceb-283a-4438-8ea6-ca3672907775 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:47:06.004959 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92faeceb-283a-4438-8ea6-ca3672907775 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:47:06.006456 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92faeceb-283a-4438-8ea6-ca3672907775 service neutron] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:47:06.016107 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1485/2961] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 16:47:05 2022] GET /identity/v3/auth/tokens => generated 3315 bytes in 36 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 16:47:06.616986 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=26 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:47:16.011828 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=6 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:48:38.368020 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d361efa-08e6-4c0a-82b6-3dcae6aca0f7 service neutron] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:48:38.370187 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-6d361efa-08e6-4c0a-82b6-3dcae6aca0f7 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:48:38.448625 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d361efa-08e6-4c0a-82b6-3dcae6aca0f7 service neutron] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:48:38.454924 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-666cbcf1-0376-42f7-93f4-ca68e4c403b6 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4e039b46c5e3445bbb39904b00c5430e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'admin', 'ResellerAdmin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:48:38.455947 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-666cbcf1-0376-42f7-93f4-ca68e4c403b6 service nova] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:48:38.456303 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-666cbcf1-0376-42f7-93f4-ca68e4c403b6 service nova] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:48:38.456661 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-666cbcf1-0376-42f7-93f4-ca68e4c403b6 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:48:38.458667 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-666cbcf1-0376-42f7-93f4-ca68e4c403b6 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:48:38.472102 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-666cbcf1-0376-42f7-93f4-ca68e4c403b6 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:48:38.473857 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-666cbcf1-0376-42f7-93f4-ca68e4c403b6 service nova] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:48:38.488926 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1477/2962] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 16:48:38 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 122 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 16:48:38.552399 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92faeceb-283a-4438-8ea6-ca3672907775 service neutron] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:48:38.582626 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92faeceb-283a-4438-8ea6-ca3672907775 service neutron] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:48:38.586659 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-907cf187-32d4-4214-ba8a-4542611b1f99 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f436f07939d4855a21b195bc7566244', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:48:38.587372 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-907cf187-32d4-4214-ba8a-4542611b1f99 service neutron] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:48:38.587653 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-907cf187-32d4-4214-ba8a-4542611b1f99 service neutron] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:48:38.587925 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-907cf187-32d4-4214-ba8a-4542611b1f99 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:48:38.589930 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-907cf187-32d4-4214-ba8a-4542611b1f99 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:48:38.604534 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-907cf187-32d4-4214-ba8a-4542611b1f99 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:48:38.605986 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-907cf187-32d4-4214-ba8a-4542611b1f99 service neutron] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:48:38.617674 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1486/2963] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 16:48:38 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 66 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 16:48:39.524972 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-666cbcf1-0376-42f7-93f4-ca68e4c403b6 service nova] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:48:39.534249 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-666cbcf1-0376-42f7-93f4-ca68e4c403b6 service nova] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:48:39.538337 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77c63f55-5e8a-4cbb-8fe9-ed4d7d407ddd service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f436f07939d4855a21b195bc7566244', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:48:39.539118 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77c63f55-5e8a-4cbb-8fe9-ed4d7d407ddd service neutron] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:48:39.539394 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77c63f55-5e8a-4cbb-8fe9-ed4d7d407ddd service neutron] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:48:39.539668 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77c63f55-5e8a-4cbb-8fe9-ed4d7d407ddd service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:48:39.541410 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-77c63f55-5e8a-4cbb-8fe9-ed4d7d407ddd service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:48:39.550833 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77c63f55-5e8a-4cbb-8fe9-ed4d7d407ddd service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:48:39.552320 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77c63f55-5e8a-4cbb-8fe9-ed4d7d407ddd service neutron] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:48:39.560858 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1478/2964] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 16:48:39 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 37 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 16:48:39.806984 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-907cf187-32d4-4214-ba8a-4542611b1f99 service neutron] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:48:39.815671 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-907cf187-32d4-4214-ba8a-4542611b1f99 service neutron] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:48:39.819807 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-271bc0fb-510b-4e13-a00d-759672ca59b5 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f436f07939d4855a21b195bc7566244', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:48:39.820401 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-271bc0fb-510b-4e13-a00d-759672ca59b5 service neutron] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:48:39.820683 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-271bc0fb-510b-4e13-a00d-759672ca59b5 service neutron] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:48:39.820958 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-271bc0fb-510b-4e13-a00d-759672ca59b5 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:48:39.822667 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-271bc0fb-510b-4e13-a00d-759672ca59b5 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:48:39.842657 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-271bc0fb-510b-4e13-a00d-759672ca59b5 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:48:39.844151 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-271bc0fb-510b-4e13-a00d-759672ca59b5 service neutron] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:48:39.856275 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1487/2965] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 16:48:39 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 50 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 16:48:49.557345 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=37 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:48:49.850316 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=21 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:48:50.712004 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77c63f55-5e8a-4cbb-8fe9-ed4d7d407ddd service neutron] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:48:50.721793 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77c63f55-5e8a-4cbb-8fe9-ed4d7d407ddd service neutron] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:48:50.725679 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-971e6b38-c7d3-46d4-bd8c-fa4687a7e7df service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f436f07939d4855a21b195bc7566244', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:48:50.726282 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-971e6b38-c7d3-46d4-bd8c-fa4687a7e7df service neutron] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:48:50.726637 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-971e6b38-c7d3-46d4-bd8c-fa4687a7e7df service neutron] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:48:50.727022 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-971e6b38-c7d3-46d4-bd8c-fa4687a7e7df service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:48:50.729161 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-971e6b38-c7d3-46d4-bd8c-fa4687a7e7df service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:48:50.793210 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-971e6b38-c7d3-46d4-bd8c-fa4687a7e7df service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:48:50.794643 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-971e6b38-c7d3-46d4-bd8c-fa4687a7e7df service neutron] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:48:50.804366 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1479/2966] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 16:48:50 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 93 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 16:48:53.862383 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-271bc0fb-510b-4e13-a00d-759672ca59b5 service neutron] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:48:53.863802 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-271bc0fb-510b-4e13-a00d-759672ca59b5 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:48:53.953098 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-271bc0fb-510b-4e13-a00d-759672ca59b5 service neutron] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:48:53.958275 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eec7e13b-0dbd-4dc3-876c-2500ed0dd1d6 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '31480b61418340bd88583f034f67412e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:48:53.958909 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eec7e13b-0dbd-4dc3-876c-2500ed0dd1d6 service placement] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:48:53.959322 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eec7e13b-0dbd-4dc3-876c-2500ed0dd1d6 service placement] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:48:53.959600 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eec7e13b-0dbd-4dc3-876c-2500ed0dd1d6 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:48:53.961294 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-eec7e13b-0dbd-4dc3-876c-2500ed0dd1d6 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:48:53.973457 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eec7e13b-0dbd-4dc3-876c-2500ed0dd1d6 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:48:53.975434 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eec7e13b-0dbd-4dc3-876c-2500ed0dd1d6 service placement] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:48:53.986652 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1488/2967] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 16:48:53 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 125 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 16:48:56.964515 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-971e6b38-c7d3-46d4-bd8c-fa4687a7e7df service neutron] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:48:56.980172 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-971e6b38-c7d3-46d4-bd8c-fa4687a7e7df service neutron] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:48:56.985985 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3be87756-8543-4d63-b647-b39af333b981 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '31480b61418340bd88583f034f67412e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:48:56.986668 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3be87756-8543-4d63-b647-b39af333b981 service placement] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:48:56.987055 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3be87756-8543-4d63-b647-b39af333b981 service placement] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:48:56.987334 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3be87756-8543-4d63-b647-b39af333b981 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:48:56.989815 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-3be87756-8543-4d63-b647-b39af333b981 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:48:57.002997 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3be87756-8543-4d63-b647-b39af333b981 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:48:57.004775 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3be87756-8543-4d63-b647-b39af333b981 service placement] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:48:57.016128 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1480/2968] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 16:48:56 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 52 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 16:48:57.038180 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eec7e13b-0dbd-4dc3-876c-2500ed0dd1d6 service placement] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:48:57.039832 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-eec7e13b-0dbd-4dc3-876c-2500ed0dd1d6 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:48:57.060935 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eec7e13b-0dbd-4dc3-876c-2500ed0dd1d6 service placement] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:48:57.067505 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebaf1c79-5929-45dd-be68-8bf92db37d06 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:48:57.068155 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebaf1c79-5929-45dd-be68-8bf92db37d06 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:48:57.068436 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebaf1c79-5929-45dd-be68-8bf92db37d06 admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:48:57.068711 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebaf1c79-5929-45dd-be68-8bf92db37d06 admin admin] PATH_INFO: `/v3/users/b6417e189b0d467480627cfd38331a9f` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:48:57.069861 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebaf1c79-5929-45dd-be68-8bf92db37d06 admin admin] RBAC: Authorizing `identity:delete_user(user_id=b6417e189b0d467480627cfd38331a9f)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:48:57.074104 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebaf1c79-5929-45dd-be68-8bf92db37d06 admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:48:57.122083 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-ebaf1c79-5929-45dd-be68-8bf92db37d06 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'b6417e189b0d467480627cfd38331a9f'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:48:57.129353 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-ebaf1c79-5929-45dd-be68-8bf92db37d06 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'b6417e189b0d467480627cfd38331a9f'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:48:57.141987 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-ebaf1c79-5929-45dd-be68-8bf92db37d06 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'b6417e189b0d467480627cfd38331a9f'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:48:57.142635 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-ebaf1c79-5929-45dd-be68-8bf92db37d06 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'b6417e189b0d467480627cfd38331a9f'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:48:57.150117 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-ebaf1c79-5929-45dd-be68-8bf92db37d06 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'b6417e189b0d467480627cfd38331a9f'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:48:57.155953 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1489/2969] 10.222.0.50 () {58 vars in 1224 bytes} [Tue Sep 13 16:48:57 2022] DELETE /identity/v3/users/b6417e189b0d467480627cfd38331a9f => generated 0 bytes in 118 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 16:48:57.325888 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3be87756-8543-4d63-b647-b39af333b981 service placement] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:48:57.338246 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3be87756-8543-4d63-b647-b39af333b981 service placement] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:48:57.396083 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7afd6bd1-8488-45dc-9516-dde6d8f3883f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:48:57.396710 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7afd6bd1-8488-45dc-9516-dde6d8f3883f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:48:57.396989 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7afd6bd1-8488-45dc-9516-dde6d8f3883f admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:48:57.397260 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7afd6bd1-8488-45dc-9516-dde6d8f3883f admin admin] PATH_INFO: `/v3/projects/02ef20a5c40945c688b9343861cadb85` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:48:57.398357 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7afd6bd1-8488-45dc-9516-dde6d8f3883f admin admin] RBAC: Authorizing `identity:delete_project(project_id=02ef20a5c40945c688b9343861cadb85)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:48:57.400782 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7afd6bd1-8488-45dc-9516-dde6d8f3883f admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:48:57.450980 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-7afd6bd1-8488-45dc-9516-dde6d8f3883f admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '02ef20a5c40945c688b9343861cadb85', 'request_id': 'req-7afd6bd1-8488-45dc-9516-dde6d8f3883f'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:48:57.462789 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-7afd6bd1-8488-45dc-9516-dde6d8f3883f admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '02ef20a5c40945c688b9343861cadb85', 'request_id': 'req-7afd6bd1-8488-45dc-9516-dde6d8f3883f'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:48:57.468871 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-7afd6bd1-8488-45dc-9516-dde6d8f3883f admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '02ef20a5c40945c688b9343861cadb85', 'request_id': 'req-7afd6bd1-8488-45dc-9516-dde6d8f3883f'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:48:57.480483 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-7afd6bd1-8488-45dc-9516-dde6d8f3883f admin admin] The token cache is being invalidate because project 02ef20a5c40945c688b9343861cadb85 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81438) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 13 16:48:57.480770 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-7afd6bd1-8488-45dc-9516-dde6d8f3883f admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:48:57.481390 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-7afd6bd1-8488-45dc-9516-dde6d8f3883f admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:48:57.482847 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1481/2970] 10.222.0.50 () {58 vars in 1233 bytes} [Tue Sep 13 16:48:57 2022] DELETE /identity/v3/projects/02ef20a5c40945c688b9343861cadb85 => generated 0 bytes in 158 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 16:48:57.486873 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebaf1c79-5929-45dd-be68-8bf92db37d06 admin admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:48:57.488312 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-ebaf1c79-5929-45dd-be68-8bf92db37d06 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:48:57.502610 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebaf1c79-5929-45dd-be68-8bf92db37d06 admin admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:48:57.559425 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d969992-9534-41e7-9b2d-ed078edc525c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:48:57.560003 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d969992-9534-41e7-9b2d-ed078edc525c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:48:57.560277 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d969992-9534-41e7-9b2d-ed078edc525c admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:48:57.560546 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d969992-9534-41e7-9b2d-ed078edc525c admin admin] PATH_INFO: `/v3/users/181b443239e8473b841ca01c13d1e446` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:48:57.561661 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d969992-9534-41e7-9b2d-ed078edc525c admin admin] RBAC: Authorizing `identity:delete_user(user_id=181b443239e8473b841ca01c13d1e446)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:48:57.563614 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d969992-9534-41e7-9b2d-ed078edc525c admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:48:57.565795 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:48:57.610800 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-1d969992-9534-41e7-9b2d-ed078edc525c admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '181b443239e8473b841ca01c13d1e446'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:48:57.619472 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-1d969992-9534-41e7-9b2d-ed078edc525c admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '181b443239e8473b841ca01c13d1e446'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:48:57.631102 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-1d969992-9534-41e7-9b2d-ed078edc525c admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '181b443239e8473b841ca01c13d1e446'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:48:57.631732 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-1d969992-9534-41e7-9b2d-ed078edc525c admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '181b443239e8473b841ca01c13d1e446'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:48:57.638799 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-1d969992-9534-41e7-9b2d-ed078edc525c admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '181b443239e8473b841ca01c13d1e446'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:48:57.644527 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1490/2971] 10.222.0.50 () {58 vars in 1224 bytes} [Tue Sep 13 16:48:57 2022] DELETE /identity/v3/users/181b443239e8473b841ca01c13d1e446 => generated 0 bytes in 158 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 16:48:57.816910 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7afd6bd1-8488-45dc-9516-dde6d8f3883f admin admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:48:57.818491 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-7afd6bd1-8488-45dc-9516-dde6d8f3883f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:48:57.834429 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7afd6bd1-8488-45dc-9516-dde6d8f3883f admin admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:48:57.844402 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:48:57.903587 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-beaea187-e5b5-4a36-abed-9247d5c454be admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:48:57.904224 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-beaea187-e5b5-4a36-abed-9247d5c454be admin admin] REQUEST_METHOD: `DELETE` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:48:57.904514 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-beaea187-e5b5-4a36-abed-9247d5c454be admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:48:57.904784 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-beaea187-e5b5-4a36-abed-9247d5c454be admin admin] PATH_INFO: `/v3/projects/da3174247d754b528afbc974e8f9bc31` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:48:57.905895 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-beaea187-e5b5-4a36-abed-9247d5c454be admin admin] RBAC: Authorizing `identity:delete_project(project_id=da3174247d754b528afbc974e8f9bc31)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:48:57.907846 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-beaea187-e5b5-4a36-abed-9247d5c454be admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:48:57.961432 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-beaea187-e5b5-4a36-abed-9247d5c454be admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'da3174247d754b528afbc974e8f9bc31', 'request_id': 'req-beaea187-e5b5-4a36-abed-9247d5c454be'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:48:57.969448 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-beaea187-e5b5-4a36-abed-9247d5c454be admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'da3174247d754b528afbc974e8f9bc31', 'request_id': 'req-beaea187-e5b5-4a36-abed-9247d5c454be'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:48:57.975156 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-beaea187-e5b5-4a36-abed-9247d5c454be admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'da3174247d754b528afbc974e8f9bc31', 'request_id': 'req-beaea187-e5b5-4a36-abed-9247d5c454be'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:48:57.989720 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-beaea187-e5b5-4a36-abed-9247d5c454be admin admin] The token cache is being invalidate because project da3174247d754b528afbc974e8f9bc31 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81438) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 13 16:48:57.990241 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-beaea187-e5b5-4a36-abed-9247d5c454be admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:48:57.991128 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-beaea187-e5b5-4a36-abed-9247d5c454be admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:48:57.992603 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1482/2972] 10.222.0.50 () {58 vars in 1233 bytes} [Tue Sep 13 16:48:57 2022] DELETE /identity/v3/projects/da3174247d754b528afbc974e8f9bc31 => generated 0 bytes in 176 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 16:49:07.641113 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=25,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:49:07.984135 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=46,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:49:13.395186 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d969992-9534-41e7-9b2d-ed078edc525c admin admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:49:13.396672 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-1d969992-9534-41e7-9b2d-ed078edc525c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:49:13.453300 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d969992-9534-41e7-9b2d-ed078edc525c admin admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:49:13.500721 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bef369d-85cd-42b1-ba01-1839fefe0701 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f436f07939d4855a21b195bc7566244', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:49:13.501319 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bef369d-85cd-42b1-ba01-1839fefe0701 service neutron] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:49:13.501597 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bef369d-85cd-42b1-ba01-1839fefe0701 service neutron] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:49:13.501871 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bef369d-85cd-42b1-ba01-1839fefe0701 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:49:13.503579 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-9bef369d-85cd-42b1-ba01-1839fefe0701 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:49:13.512817 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bef369d-85cd-42b1-ba01-1839fefe0701 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:49:13.514241 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bef369d-85cd-42b1-ba01-1839fefe0701 service neutron] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:49:13.522343 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1491/2973] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 16:49:13 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 128 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 16:49:13.641057 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-beaea187-e5b5-4a36-abed-9247d5c454be admin admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:49:13.642644 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-beaea187-e5b5-4a36-abed-9247d5c454be admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:49:13.655437 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-beaea187-e5b5-4a36-abed-9247d5c454be admin admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:49:13.659268 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cdd4b1e-52b4-4824-8699-c660e1ef1288 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f436f07939d4855a21b195bc7566244', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:49:13.659934 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cdd4b1e-52b4-4824-8699-c660e1ef1288 service neutron] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:49:13.660213 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cdd4b1e-52b4-4824-8699-c660e1ef1288 service neutron] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:49:13.660486 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cdd4b1e-52b4-4824-8699-c660e1ef1288 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:49:13.662236 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-0cdd4b1e-52b4-4824-8699-c660e1ef1288 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:49:13.675625 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0cdd4b1e-52b4-4824-8699-c660e1ef1288 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:49:13.677052 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0cdd4b1e-52b4-4824-8699-c660e1ef1288 service neutron] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:49:13.685506 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1483/2974] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 16:49:13 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 45 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 16:49:13.788125 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bef369d-85cd-42b1-ba01-1839fefe0701 service neutron] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:49:13.790402 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-9bef369d-85cd-42b1-ba01-1839fefe0701 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:49:13.801441 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bef369d-85cd-42b1-ba01-1839fefe0701 service neutron] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:49:13.805124 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09476ad9-11be-4031-bf98-19912936788c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f436f07939d4855a21b195bc7566244', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:49:13.805834 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09476ad9-11be-4031-bf98-19912936788c service neutron] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:49:13.806117 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09476ad9-11be-4031-bf98-19912936788c service neutron] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:49:13.806393 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09476ad9-11be-4031-bf98-19912936788c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:49:13.808123 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-09476ad9-11be-4031-bf98-19912936788c service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:49:13.878832 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09476ad9-11be-4031-bf98-19912936788c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:49:13.880349 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09476ad9-11be-4031-bf98-19912936788c service neutron] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:49:13.891953 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1492/2975] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 16:49:13 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 104 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 16:49:13.956904 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cdd4b1e-52b4-4824-8699-c660e1ef1288 service neutron] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:49:13.958307 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-0cdd4b1e-52b4-4824-8699-c660e1ef1288 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:49:14.030369 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cdd4b1e-52b4-4824-8699-c660e1ef1288 service neutron] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:49:14.035285 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7788f4a7-0aff-438e-8536-aa556900191f service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '31480b61418340bd88583f034f67412e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:49:14.035883 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7788f4a7-0aff-438e-8536-aa556900191f service placement] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:49:14.036187 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7788f4a7-0aff-438e-8536-aa556900191f service placement] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:49:14.036454 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7788f4a7-0aff-438e-8536-aa556900191f service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:49:14.038159 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-7788f4a7-0aff-438e-8536-aa556900191f service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:49:14.050110 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7788f4a7-0aff-438e-8536-aa556900191f service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:49:14.051714 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7788f4a7-0aff-438e-8536-aa556900191f service placement] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:49:14.061854 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1484/2976] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 16:49:13 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 105 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 16:49:16.027322 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09476ad9-11be-4031-bf98-19912936788c service neutron] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:49:16.028974 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-09476ad9-11be-4031-bf98-19912936788c service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:49:16.050255 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09476ad9-11be-4031-bf98-19912936788c service neutron] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:49:16.062666 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-021b2dbd-f9a0-4235-a590-19ca472755d9 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4e039b46c5e3445bbb39904b00c5430e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'admin', 'ResellerAdmin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:49:16.063402 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-021b2dbd-f9a0-4235-a590-19ca472755d9 service nova] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:49:16.063682 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-021b2dbd-f9a0-4235-a590-19ca472755d9 service nova] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:49:16.063956 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-021b2dbd-f9a0-4235-a590-19ca472755d9 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:49:16.066383 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-021b2dbd-f9a0-4235-a590-19ca472755d9 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:49:16.082005 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-021b2dbd-f9a0-4235-a590-19ca472755d9 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:49:16.083767 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-021b2dbd-f9a0-4235-a590-19ca472755d9 service nova] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:49:16.096037 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1493/2977] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 16:49:16 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 69 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 16:49:24.057554 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=38 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:49:26.090792 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=86 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:50:44.854131 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7788f4a7-0aff-438e-8536-aa556900191f service placement] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:50:44.855839 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-7788f4a7-0aff-438e-8536-aa556900191f service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:50:44.921448 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7788f4a7-0aff-438e-8536-aa556900191f service placement] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:50:44.926074 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf87e1b2-8111-4b8b-bea5-d45155fe2184 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:50:44.926703 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf87e1b2-8111-4b8b-bea5-d45155fe2184 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:50:44.927018 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf87e1b2-8111-4b8b-bea5-d45155fe2184 admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:50:44.927298 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf87e1b2-8111-4b8b-bea5-d45155fe2184 admin admin] PATH_INFO: `/v3/users/0e8f8e2bf7f440f291b47dfd9357bc14` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:50:44.928436 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf87e1b2-8111-4b8b-bea5-d45155fe2184 admin admin] RBAC: Authorizing `identity:delete_user(user_id=0e8f8e2bf7f440f291b47dfd9357bc14)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:50:44.930332 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf87e1b2-8111-4b8b-bea5-d45155fe2184 admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:50:44.975758 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-bf87e1b2-8111-4b8b-bea5-d45155fe2184 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '0e8f8e2bf7f440f291b47dfd9357bc14'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:50:44.982809 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-bf87e1b2-8111-4b8b-bea5-d45155fe2184 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '0e8f8e2bf7f440f291b47dfd9357bc14'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:50:44.995706 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-bf87e1b2-8111-4b8b-bea5-d45155fe2184 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '0e8f8e2bf7f440f291b47dfd9357bc14'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:50:44.996291 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-bf87e1b2-8111-4b8b-bea5-d45155fe2184 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '0e8f8e2bf7f440f291b47dfd9357bc14'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:50:45.003420 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-bf87e1b2-8111-4b8b-bea5-d45155fe2184 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '0e8f8e2bf7f440f291b47dfd9357bc14'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:50:45.009611 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1485/2978] 10.222.0.50 () {58 vars in 1224 bytes} [Tue Sep 13 16:50:44 2022] DELETE /identity/v3/users/0e8f8e2bf7f440f291b47dfd9357bc14 => generated 0 bytes in 156 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 16:50:45.188193 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-021b2dbd-f9a0-4235-a590-19ca472755d9 service nova] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:50:45.198808 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-021b2dbd-f9a0-4235-a590-19ca472755d9 service nova] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:50:45.258296 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5750f9f9-c858-4121-a942-27298f56c719 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:50:45.258853 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5750f9f9-c858-4121-a942-27298f56c719 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:50:45.259163 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5750f9f9-c858-4121-a942-27298f56c719 admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:50:45.259444 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5750f9f9-c858-4121-a942-27298f56c719 admin admin] PATH_INFO: `/v3/projects/971ad6e8cfa54529a3e6aac596203515` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:50:45.260545 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5750f9f9-c858-4121-a942-27298f56c719 admin admin] RBAC: Authorizing `identity:delete_project(project_id=971ad6e8cfa54529a3e6aac596203515)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:50:45.262483 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5750f9f9-c858-4121-a942-27298f56c719 admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:50:45.308832 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-5750f9f9-c858-4121-a942-27298f56c719 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '971ad6e8cfa54529a3e6aac596203515', 'request_id': 'req-5750f9f9-c858-4121-a942-27298f56c719'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:50:45.317191 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-5750f9f9-c858-4121-a942-27298f56c719 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '971ad6e8cfa54529a3e6aac596203515', 'request_id': 'req-5750f9f9-c858-4121-a942-27298f56c719'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:50:45.323465 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-5750f9f9-c858-4121-a942-27298f56c719 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '971ad6e8cfa54529a3e6aac596203515', 'request_id': 'req-5750f9f9-c858-4121-a942-27298f56c719'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:50:45.334739 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-5750f9f9-c858-4121-a942-27298f56c719 admin admin] The token cache is being invalidate because project 971ad6e8cfa54529a3e6aac596203515 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81437) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 13 16:50:45.335068 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-5750f9f9-c858-4121-a942-27298f56c719 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:50:45.335707 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-5750f9f9-c858-4121-a942-27298f56c719 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:50:45.337204 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1494/2979] 10.222.0.50 () {58 vars in 1233 bytes} [Tue Sep 13 16:50:45 2022] DELETE /identity/v3/projects/971ad6e8cfa54529a3e6aac596203515 => generated 0 bytes in 150 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 16:50:45.341414 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf87e1b2-8111-4b8b-bea5-d45155fe2184 admin admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:50:45.343506 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-bf87e1b2-8111-4b8b-bea5-d45155fe2184 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:50:45.356421 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf87e1b2-8111-4b8b-bea5-d45155fe2184 admin admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:50:45.415293 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17e7dbd0-86a2-4345-ac31-17bcc9f31b24 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:50:45.415930 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17e7dbd0-86a2-4345-ac31-17bcc9f31b24 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:50:45.416208 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17e7dbd0-86a2-4345-ac31-17bcc9f31b24 admin admin] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:50:45.416482 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17e7dbd0-86a2-4345-ac31-17bcc9f31b24 admin admin] PATH_INFO: `/v3/users/f586ba17a8e24707bfc02790060a2a4c` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:50:45.417618 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17e7dbd0-86a2-4345-ac31-17bcc9f31b24 admin admin] RBAC: Authorizing `identity:delete_user(user_id=f586ba17a8e24707bfc02790060a2a4c)` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:50:45.419580 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17e7dbd0-86a2-4345-ac31-17bcc9f31b24 admin admin] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:50:45.442808 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:SELECT=86,keystone:DELETE=13,keystone:INSERT=1 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:50:45.470469 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-17e7dbd0-86a2-4345-ac31-17bcc9f31b24 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'f586ba17a8e24707bfc02790060a2a4c'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:50:45.478830 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-17e7dbd0-86a2-4345-ac31-17bcc9f31b24 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'f586ba17a8e24707bfc02790060a2a4c'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:50:45.491495 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-17e7dbd0-86a2-4345-ac31-17bcc9f31b24 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'f586ba17a8e24707bfc02790060a2a4c'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:50:45.492189 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-17e7dbd0-86a2-4345-ac31-17bcc9f31b24 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'f586ba17a8e24707bfc02790060a2a4c'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:50:45.499917 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.notifications [None req-17e7dbd0-86a2-4345-ac31-17bcc9f31b24 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'f586ba17a8e24707bfc02790060a2a4c'} {{(pid=81438) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:50:45.506074 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1486/2980] 10.222.0.50 () {58 vars in 1224 bytes} [Tue Sep 13 16:50:45 2022] DELETE /identity/v3/users/f586ba17a8e24707bfc02790060a2a4c => generated 0 bytes in 165 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 16:50:45.682541 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5750f9f9-c858-4121-a942-27298f56c719 admin admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:50:45.684091 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-5750f9f9-c858-4121-a942-27298f56c719 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:50:45.699315 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5750f9f9-c858-4121-a942-27298f56c719 admin admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:50:45.758688 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d0d6fdc-f443-4c6a-865d-ad277e4a5bca admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a915a0eb3c42c7af97dcc7eed5b42f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '555bee08a3fa4d08bdd8ecc881adc45b', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:50:45.759346 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d0d6fdc-f443-4c6a-865d-ad277e4a5bca admin admin] REQUEST_METHOD: `DELETE` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:50:45.759680 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d0d6fdc-f443-4c6a-865d-ad277e4a5bca admin admin] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:50:45.759902 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d0d6fdc-f443-4c6a-865d-ad277e4a5bca admin admin] PATH_INFO: `/v3/projects/2e5f6804a4f4475d8580763fe4029ea5` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:50:45.761004 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d0d6fdc-f443-4c6a-865d-ad277e4a5bca admin admin] RBAC: Authorizing `identity:delete_project(project_id=2e5f6804a4f4475d8580763fe4029ea5)` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:50:45.762936 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d0d6fdc-f443-4c6a-865d-ad277e4a5bca admin admin] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:50:45.799150 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=90,keystone:DELETE=9,keystone:INSERT=1 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:50:45.813753 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-5d0d6fdc-f443-4c6a-865d-ad277e4a5bca admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '2e5f6804a4f4475d8580763fe4029ea5', 'request_id': 'req-5d0d6fdc-f443-4c6a-865d-ad277e4a5bca'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:50:45.821000 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-5d0d6fdc-f443-4c6a-865d-ad277e4a5bca admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '2e5f6804a4f4475d8580763fe4029ea5', 'request_id': 'req-5d0d6fdc-f443-4c6a-865d-ad277e4a5bca'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:50:45.827384 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-5d0d6fdc-f443-4c6a-865d-ad277e4a5bca admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '2e5f6804a4f4475d8580763fe4029ea5', 'request_id': 'req-5d0d6fdc-f443-4c6a-865d-ad277e4a5bca'} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:50:45.837896 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-5d0d6fdc-f443-4c6a-865d-ad277e4a5bca admin admin] The token cache is being invalidate because project 2e5f6804a4f4475d8580763fe4029ea5 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81437) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 13 16:50:45.838184 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-5d0d6fdc-f443-4c6a-865d-ad277e4a5bca admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:50:45.838821 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.notifications [None req-5d0d6fdc-f443-4c6a-865d-ad277e4a5bca admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81437) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 16:50:45.840347 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1495/2981] 10.222.0.50 () {58 vars in 1233 bytes} [Tue Sep 13 16:50:45 2022] DELETE /identity/v3/projects/2e5f6804a4f4475d8580763fe4029ea5 => generated 0 bytes in 158 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 16:50:54.671449 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17e7dbd0-86a2-4345-ac31-17bcc9f31b24 admin admin] Authenticating user token {{(pid=81438) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:50:54.673028 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-17e7dbd0-86a2-4345-ac31-17bcc9f31b24 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:50:54.755052 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17e7dbd0-86a2-4345-ac31-17bcc9f31b24 admin admin] Validating token access rules against request {{(pid=81438) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:50:54.813748 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-543d3075-3fd1-4c52-b786-3d1180d664aa service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '31480b61418340bd88583f034f67412e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81438) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:50:54.814351 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-543d3075-3fd1-4c52-b786-3d1180d664aa service placement] REQUEST_METHOD: `GET` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:50:54.814627 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-543d3075-3fd1-4c52-b786-3d1180d664aa service placement] SCRIPT_NAME: `/identity` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:50:54.814921 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.server.flask.request_processing.req_logging [None req-543d3075-3fd1-4c52-b786-3d1180d664aa service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81438) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:50:54.816629 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.fernet_utils [None req-543d3075-3fd1-4c52-b786-3d1180d664aa service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81438) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:50:54.828372 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-543d3075-3fd1-4c52-b786-3d1180d664aa service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:50:54.830192 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-543d3075-3fd1-4c52-b786-3d1180d664aa service placement] RBAC: Authorization granted {{(pid=81438) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:50:54.840052 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: [pid: 81438|app: 0|req: 1487/2982] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 16:50:54 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 170 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 16:50:55.834795 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG dbcounter [-] [81437] Writing DB stats keystone:SELECT=12,keystone:DELETE=5,keystone:INSERT=1 {{(pid=81437) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:51:04.835770 dv-cinder-820027-28-iscsi devstack@keystone.service[81438]: DEBUG dbcounter [-] [81438] Writing DB stats keystone:DELETE=7,keystone:SELECT=83,keystone:INSERT=1 {{(pid=81438) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 16:56:00.609490 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d0d6fdc-f443-4c6a-865d-ad277e4a5bca admin admin] Authenticating user token {{(pid=81437) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 16:56:00.610931 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.fernet_utils [None req-5d0d6fdc-f443-4c6a-865d-ad277e4a5bca admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81437) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 16:56:00.650795 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d0d6fdc-f443-4c6a-865d-ad277e4a5bca admin admin] Validating token access rules against request {{(pid=81437) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 16:56:00.656130 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e92d69b-5bd9-48a9-b2eb-73a4db080c38 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '31480b61418340bd88583f034f67412e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5811e3b83a1e462e9320cad79a3ad791', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81437) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 16:56:00.656731 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e92d69b-5bd9-48a9-b2eb-73a4db080c38 service placement] REQUEST_METHOD: `GET` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 16:56:00.657013 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e92d69b-5bd9-48a9-b2eb-73a4db080c38 service placement] SCRIPT_NAME: `/identity` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 16:56:00.657289 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e92d69b-5bd9-48a9-b2eb-73a4db080c38 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81437) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 16:56:00.666227 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e92d69b-5bd9-48a9-b2eb-73a4db080c38 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 16:56:00.667879 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e92d69b-5bd9-48a9-b2eb-73a4db080c38 service placement] RBAC: Authorization granted {{(pid=81437) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 16:56:00.677785 dv-cinder-820027-28-iscsi devstack@keystone.service[81437]: [pid: 81437|app: 0|req: 1496/2983] 10.222.0.50 () {60 vars in 1365 bytes} [Tue Sep 13 16:56:00 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 69 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0)